Accessing remoting exceptions and original causes of Exceptions in Service Fabric stateless/stateful services in v3.0+/v6.1+

The section "Remoting Exception Handling" on this docs.microsoft.com page appears to be the sum and total of the Service Fabric team's documentation on proper exception handling for Service Fabric remoting. It's two paragraphs. And it's completely insufficient bullshit.

What they fail to explain and be explicit about is the fact that Service Fabric takes WCF's exception handling capabilities to the extreme and handles all exceptions automatically by serializing them using DataContractSerializer, remoting them back to the caller, deserializing them on the caller and converting them back to .NET exceptions and making them accessible via the AggregateException.InnerException property in a try-catch block around a ServiceProxy / ActorProxy.Create<>() result's service method. Well, that's all well and good, except for the fact that according to multiple github.com issues for Service Fabric, the Service Fabric team broke the shit out of this nice facility in V2 remoting. So, now, we get to revert somewhat back to WCF's exception handling model, in the earlier days of WCF, and throw a FaultException<MyFault> where MyFault is your own custom DataContract serializable data contract object. Fantastic job boys and girls. When are you going to grow into your big person pants and fucking test things properly before releasing them ? #GettingSickAndFuckingTiredOfLazyAssUndiligentMillenialDevelopers

Troubleshooting connections to a service running in Service Fabric in Azure

Recently I decided to try deploying a Service Fabric cluster to Azure and investigate what it takes to create applications with Service Fabric. I used the default ARM template to deploy the cluster, with the parameters in the parameter file set appropriately. I've been able to successfully deploy the cluster itself, along with a private sample application that's showing as running and healthy within the cluster. (all of this with VSTS, but that's for another post). I'm now running into the problem of actually connecting to the Service.

When I use postman to connect to the service, I'm connecting to a URL like :

https://mycluster.westus.cloudapp.azure.com:8870/api/things

However, when I send my request, I instantly see Postman fail to connect:

Here's the steps taken when verifying all the settings so far:

• I got the address for my cluster from the "IP Address" resource that came with the ARM template that's in the Azure portal, using the "Copy" functionality. 
• I've verified that the Load Balancer that was set up by the ARM template is correctly configured to use my application ports.
• I've consulted the documentation for Load Balancer health probes here to ensure that my machines are using the correct type of probe: https://docs.microsoft.com/en-ca/azure/load-balancer/load-balancer-custom-probe-overview#learn-about-the-types-of-probes . In my case, I don't want to use HTTP even though I've got an HTTP service because I'd need a 200 response. Instead, we use the more basic TCP probe which determines health status based on TCP handshake, which should be just fine.
• I've updated the Diagnostics lettings on the Load Balancer to pipe logs out to a storage account. Using the generated output, I've found that my health probe contradicts my expectations and is in fact failing. What's worse, there's a timing issue: the health probe fails too many times before the Service Fabric Host can startup on the VMs and then permanently marks the hosts as failed, preventing accessibility to any of my VMs, which would seem to explain my inability to connect to my services AND the speed with which the response is returned (because the traffic doesn't even get past the Load Balancer).
• I've used Remote Desktop to gain access to the Virtual Machine Scale Set VMs thanks to the default settings that came with the Service Fabric ARM template. Loading up PowerShell and executing the command "iwr -Method Get -Uri https://localhost:8870/api/Things" yields the error "iwr : The underlying connection was closed: An unexpected error occurred on a send". It would seem that I can't even get an actual connection to my service working on the local machine. This would explain why the health probes are failing: they're perfectly legit.
• Running the command "netstat -an | ? { $_ -like '*8870*' }" on the VMSS VM indicates that the Service Fabric Host has in fact launched my process on my expected port of 8870 and the process is listening on that port. Curiously, I'm also seeing an established connection on that port as well. This is at least somewhat consistent with the fact that the Service Fabric Management Portal is showing my service as healthy on all nodes, but inconsistent with the status of the health probe.
• Figuring that I now have a past problem that I already solved, I tried setting the permissions of the certificate stores for the certificates my API application uses. After some time waiting for the load balancer health probes to update, they were now able to connect and the services were running properly.

Creating Azure Services with Visual Studio 2013 and Windows 8.1

1. Download and install Visual Studio 2013
2. Download and install the Web Platform Installer, v5.0 or greater
3. From the Web Platform Installer, install:
1. Windows Azure SDK and related Powershell utilities and command line tools
4. Start a new solution (or open an existing one)
5. From the NuGet package manager in your solution, ensure that you've installed the 'WindowsAzure.Storage' package, or at least have it in your cache. This is going to be required by the New Project wizard when generating the project.
6. Add a new project
7. In the 'Add new project' wizard, select the C# projects -> Cloud -> Windows Azure Cloud Service
8. Follow this absurdly easy tutorial on implementing an ErrorHandler interceptor (behavior)
9. Add logging to your application using the Microsoft Patterns & Practices Enterprise Library Logging Application Block (which can be found here). 
10. Create a SQL server database in Azure using this tutorial on MSDN. When designing the user roles and authentication, it's recommended that you use the ASP.NET Identity membership design and design your tables accordingly around this.
1. Microsoft has provided an extension to the ASP.NET Identity membership framework specifically for EntityFramework. They recommend that you use a code-first model for generating entities.
2. Log in to Azure through the management portal: http://manage.windowsazure.com/
3. Configure all of your connections and permissions to the database. e.g. you may want to have multiple users: one for read-only operations, another for read-write operations.
4. Generate or write your data model. This article will show you how to create a code-first data model with Entity Framework. Ensure that you include users so that you can support proper application authentication and authorization via the ASP.NET Identity membership framework.
5. The link above also includes instructions on using code-first migrations for when you update your data model.
6. TODO: Elaborate on how to properly set up the database when performing a code-first database design
11. Add an IoC container to your WCF service to enable you to easily develop and unit test it. I've chosen to use Ninject because despite a few minutes of inital frustration, it's actually exceedingly easy to integrate with WCF, especially now that there's the Ninject WCF extensions NuGet package. To integration Ninject with your WCF service, perform the following steps:
1. Install-Package Ninject.Extensions.Wcf -Version 3.2.1.0 (Ninject and Ninject.Web.Common are installed as dependencies for you automatically).
2. Create a NinjectModule descendant to bind your interfaces to concrete implementations. It should look something like the following:

   namespace MyService.CloudStorage.Support
   {
       using System.Diagnostics.CodeAnalysis;
       using System.ServiceModel;
       using Common.Interfaces;
       using global::Ninject.Modules;
       using global::Ninject.Syntax;
       using Services;
       using NinjectServiceHost = global::Ninject.Extensions.Wcf.NinjectServiceHost;
   
       /// <summary>
       /// A <see cref="NinjectModule"/> descendant for bootstrapping our application
       /// </summary>
       public class MyServiceCloudStorageNinjectModule : NinjectModule
       {
           /// <inheritdoc/>
           [SuppressMessage("Microsoft.StyleCop.CSharp.DocumentationRules", "SA1604:ElementDocumentationMustHaveSummary", Justification = "InheritDoc")]
           public override void Load()
           {
               this.Bind<IResolutionRoot>().ToConstant(Kernel);
               this.Bind<ServiceHost>().To<NinjectServiceHost>();
               this.Bind<IMyServiceStorage>().To<MyServiceStorage>();
               this.Bind<ILoggingService>().To<MicrosoftEnterpriseLoggingBlockLoggingService>();
           }
       }
   }
   

3. Create a Global Application Class (global.asax) if one's not already created: Right-click on your project -> Add -> New Item ... and in the window that comes up, go to Visual C# -> Web -> Global Application File

4. Update your Global class to extend from Ninject.Web.Common.NinjectHttpApplication and override the CreateKernel() method and return a new CustomNinjectModule (as created above). The method should look something like this:

   namespace MyService.CloudStorage
   {
       using System.Diagnostics.CodeAnalysis;
       using System.Web;
       using Ninject;
       using Ninject.Web.Common;
       using Support;
   
       /// <summary>
       /// A global <see cref="HttpApplication"/> class for managing the lifecycle
       /// of the application
       /// </summary>
       public class Global : NinjectHttpApplication
       {
           /// <inheritdoc/>
           [SuppressMessage("Microsoft.StyleCop.CSharp.DocumentationRules", "SA1604:ElementDocumentationMustHaveSummary", Justification = "InheritDoc")]
           [SuppressMessage("Microsoft.StyleCop.CSharp.DocumentationRules", "SA1615:ElementReturnValueMustBeDocumented", Justification = "InheritDoc")]
           protected override IKernel CreateKernel()
           {
               return new StandardKernel(new NinjectSettings(), new MyServiceCloudStorageNinjectModule());
           }
    }
   }
   

1. Update your .svc file for your service(s) and add an extra XML attribute that looks like this: <%@ ServiceHost Service="HelloNinjectWcf.Service.GreetingService" Factory="Ninject.Extensions.Wcf.NinjectServiceHostFactory" %>

12. Implement your business logic
1. If you're designing your service properly, you'll need to ensure that communications between clients and your service are secure. Toward that end, you'll need to use SSL to encrypt your communications. There's a number of things involved in this:
1. Generate certificates for your server and your client
2. Ensure that your IIS server is correctly configured to use 'https' bindings on your site, along with the server-side certificate that you've generated.
3. If you're using Windows Store Apps to access a WCF service, you'll need to ensure that you use a CustomBinding correctly configured with the right *BindingElement objects to create an SSL secured, HTTPS-transported binding.
• For the server, in your service's concrete implementation, you'll need to remove any .config file configuration, and add a method specified according to a convention that WCF recognizes that looks like the following:

  /// <summary>
  /// The service certificate store name
  /// </summary>
  private const StoreName ServiceCertificateStoreName = StoreName.My;
  
  /// <summary>
  /// The service certificate store location
  /// </summary>
  private const StoreLocation ServiceCertificateStoreLocation = StoreLocation.LocalMachine;
  
  /// <summary>
  /// Configures the specified configuration.
  /// </summary>
  /// <param name="config">The configuration.</param>
  /// <remarks>
  /// A service endpoint configuration method determined by convention.
  /// <see href="http://msdn.microsoft.com/en-us/library/hh205277(v=vs.110).aspx"/>
  /// </remarks>
  public static void Configure(ServiceConfiguration config)
  {
   ServiceEndpoint serviceEndpoint = new ServiceEndpoint(
    ContractDescription.GetContract(typeof(IMyServiceStorage), typeof(MyServiceStorage)), 
    new CustomBinding(
     new TransportSecurityBindingElement(),
     new SslStreamSecurityBindingElement { RequireClientCertificate = false },
     new TextMessageEncodingBindingElement(MessageVersion.Soap12WSAddressing10, Encoding.UTF8),
     new HttpsTransportBindingElement()
    ), 
    new EndpointAddress("https://localhost/MyService.CloudStorage/MyServiceStorage.svc")
   );
  
   config.AddServiceEndpoint(serviceEndpoint);
  
   const string ServiceCertificateThumbprint = "[a 40 digit hexadecimal certificate thumbprint here]";
  
   X509Store certificateStore = new X509Store(ServiceCertificateStoreName, ServiceCertificateStoreLocation);
  
   certificateStore.Open(OpenFlags.ReadOnly);
  
   X509Certificate2Collection x509Certificate2Collection = certificateStore.Certificates.Find(
    findType: X509FindType.FindByThumbprint,
    findValue: ServiceCertificateThumbprint,
    validOnly: false
    );
  
   certificateStore.Close();
  
   X509Certificate2 serviceCertificate = x509Certificate2Collection.Cast<X509Certificate2>().FirstOrDefault();
  
   if (serviceCertificate == null)
   {
    throw new ConfigurationErrorsException(String.Format("No certificate representing the service with thumbprint {0} could be found in {1} store at {2} location", ServiceCertificateThumbprint, ServiceCertificateStoreName, ServiceCertificateStoreLocation));
   }
  
   ServiceCredentials serviceCredentials = new ServiceCredentials
   {
    IdentityConfiguration = new IdentityConfiguration
    {
     // TODO: Change this to authenticate the clients
     CertificateValidationMode = X509CertificateValidationMode.None
    },
    ServiceCertificate =
    {
     Certificate = serviceCertificate
    }/*,
    ClientCertificate =
    {
     // TODO: Resolve the client certificate
     Certificate = serviceCertificate
    }*/
   };
  
   config.Description.Behaviors.Add(serviceCredentials);
   // config.Description.Behaviors.Add(new ServiceMetadataBehavior { HttpGetEnabled = true, HttpsGetEnabled = true });
  }
  



• For the Windows Store App client, you'll need to have a similarly configured counterpart channel factory and binding for connecting to the service:

  this.channelFactory = new ChannelFactory<IMyServiceStorageChannel>(
   binding: new CustomBinding(
    new TransportSecurityBindingElement(),
    new SslStreamSecurityBindingElement(),
    new TextMessageEncodingBindingElement(MessageVersion.Soap12WSAddressing10, Encoding.UTF8),
    new HttpsTransportBindingElement()
   ),
   remoteAddress: new EndpointAddress(serviceUri)
  );
  
  this.passwordVault = new PasswordVault();
  
  // This IDispatchMessageInspector is a custom addition for our own brand of authentication
  this.channelFactory.Endpoint.EndpointBehaviors.Add(new ClientAuthenticationDispatchMessageInspector(this.passwordVault));
  

• For the Windows Store App, you'll also need to have the server's public key (assuming it's not trusted, e.g. a self-signed certificate) added to the app's certificate declarations in the package manifest. There's a video on how to do it here on Channel 9. For the sake of convenience, I'll reproduce the steps here:
1. Obtain your server's public key in DER-encoded .cer format.
2. Open the Package.appxmanifest file for your App in Visual Studio
3. Go to the Declarations tab.
4. Under the 'Available Declarations' box, select 'Certificates' and click 'Add' if there's no Certificates declaration already added.
5. Select the 'Certificates' declaration in the 'Supported Declarations' box.
6. In the 'Certificates' group on the large pane, click 'Add New'.
7. In the certificate parameters box that comes up, enter 'Root' in the 'Name' field, and select your public key file. Once you do this, Visual Studio will automatically import it into your project.
8. Save the manifest.
13. Publish it to Azure Services
1. If you've designed your application correctly, you'll be using HTTPS for communicating with your clients. Read Microsoft's guide on MSDN to uploading a certificate with your service.
14. Implement your Windows 8.1 client // TODO: Elaborate on this
15. Unit test your Windows 8.1 client on your own local machine.
1. Visual Studio 2012 / .NET 4.5 added the ability to do asynchronous unit tests to MSTest!
2. In order to unit test the Windows 8.1 Metro client against services on your localhost, you'll have to read this. It describes the new security features in Windows 8(.1) and how to explicitly enable your application to communicate through the network loopback interface. 
3. You can also read this. Bottom line, you have to ensure that you enable a loopback exemption for your unit tests.
16. Unit / integration test it against your Azure service
1. Ensure that you've created a 'Staging' area in your Azure configuration panel and you're not testing against production! Testing against production is extremely poor practice!
17. Publish your App on the Windows Store if you so choose

Journey to robust windows services: killing a service that failed during OnStartup and now won't stop (even though Stop() has been called)

I've found that sometimes I don't exactly get my service configuration right the first time, and this can cause a failure when calling OnStartup. As a result, when I call Stop() during the exception handling in OnStartup(), the service gets stuck in the 'Stopping' state in the services manager. Here's a quick command line to help you out with that :

TaskKill /F /FI "Services eq [service name as diplayed in the service's properties]"

This will get your process knocked off so that you can resume debugging and building.

Journey to robust web services: Fixing an error 404.3 message in IIS 7.0 / 8.0

I've recently had to go out of town, and only had my personal laptop to take with me. I usually develop in a Windows 7 environment with IIS 7.0 / IIS Express 7.5, but my laptop is Windows 8, so of course I have to go through the process of setting up my development environment all over again. One of the common errors I see when setting up a new environment is error 404.3: Not found. I run into it over and over again and when I fix it, it seems like such a trivial thing and I never remember to write down the fix to reduce my setup time in the future. Today that changes. The problem can be caused by a couple of things:
1) You don't have the application setup in IIS. You need to setup the application in IIS, *and* you need to point it to your web project directory.
2) You don't have all the requisite components installed in IIS. In order to properly install WCF, you must have IIS installed, you must have ASP.NET installed, and you must have WCF registered. Up to Windows 7 / .NET 4.0, this means you must have, at some point, run aspnet_regiis.exe -ir. On Windows 8, this means you must have installed ASP.NET 3.5 / 4.5, and installed HTTP activation for each (as well as corresponding activations for any other protocols such as net.tcp.

Journey to robust windows services: Service could not be installed. Verify that you have sufficient privileges to install system services

This is an annoying and almost entirely useless error. It can be caused by any number of things. However, in my case today, it was caused by the fact that the EventLog Source hadn't been created when I tried to install my Windows Service using the WiX installer that I had developed (which it turned out wasn't quite complete). Fortunately, WiX (at least as of 3.5) has built-in support for creating Windows Event Log sources as part of the installation process. Fortunately, I was able to rely on this article on stackoverflow.com to guide me through the process. The gist of the article:

1. Add the "http://schemas.microsoft.com/wix/UtilExtension" namespace to your .wxs file root XML element
2. Add a util:EventSource Log="Application" Name="MyServiceLogSourceName" EventMessageFile="%SystemRoot%\Microsoft.NET\Framework\v2.0.50727\EventLogMessages.dll" element to each component which requires an Event Log source
3. Problem solved

Journey to robust windows services: debugging your WiX custom actions

I've recently had some trouble getting WiX custom actions to work, and after searching around for a while, I found this article. It gives the reader two options for how to debug WiX custom actions, but I had to use both in combination to get it to work. I'll repeat the steps here, just in case the link goes stale:

1. Follow all the necessary steps to create a WiX setup project, along with a separate Custom Action project (available as a project type in the New Project dialog in Visual Studio 2010+ with WiX 3.7)
2. After you've created your custom action method and gotten the project building, add System.Diagnostics.Debugger.Launch(); at the beginning of your custom action method. This will kick the custom action into the debugger so that you can debug it (assuming you've got the wixpdb in the same directory from which you launched the installer).
3. Go to Control Panel -> System -> Advanced System Settings -> Advanced (tab) -> Environment Settings (button) -> System Variables (group) -> New .... (button) 
4. In the dialog that comes up, create a new variable called 'MMsiBreak' (without the quotes) and give it the value of your custom action method (e.g. MyCustomActionMethod)
5. Now run the debug version of your installer and it should get kicked right into your method and allow you to debug your custom action.

Journey to robust web services: debugging services and logging messages

One of the quickest and best ways to debug issues with the services you're developing is to log and trace all activity with WCF along with the messages being sent back and forth. This page on MSDN will provide you with the instructions required to setup logging and tracking for WCF.

Journey to robust web services: my clients are unable to connect!

I've been testing out some web services I've been working on extensively on my own machine, but the time has finally come where I want to start making clients that aren't running on the same machine. In one case, I have a WCF service running on Windows 7, but the client I want to connect to that service is running on Windows 8 on a separate machine. When I tried to connect, I would get a variety of errors, the most common of which being that the socket on the server was forcibly closed. I Googled around a bit, and the most common cause of this problem that I found was that the server's quota's were being reached and therefore the client was being rejected, however this shouldn't have been the case for me since there was no way my quota's could have been reached. After further research, I found that since I don't want security (yet), I had to apply some settings in order for clients that weren't on my localhost to connect. In order to allow external clients to connect, I had to change the following :

1. In my , I had to change my base address to use the network name of my computer (rather than localhost) in the base address URL.
2. In the net.tcp and ws2007HttpBindings for my services, I had to go to the the child element of the elements, and explicitly set mode="None" on those elements, otherwise they may assume some form of security by default depending on the type of the binding.

Now, I did mention above that I don't want to add security (*YET*) being the important qualifier of that statement. Security is of course always paramount, and should always be baked into a product as early as possible. As soon as I get to it, I'll add a post about all the steps required to add security to your WCF clients and services, along with links to the relevant MSDN articles (and any other useful ones that I may find.

Journey to robust web services: debugging a net.tcp service

I'm creating a net.tcp-based WCF web service for use with my current project, and so far the learning curve hasn't been very shallow. I've just recently learned, thanks to this post on Stack Overflow, that the default ASP.NET application doesn't support the net.tcp protocol, so IIS must be used instead. With that in mind, I've decided to move on to just using the Web Service I'm creating through a Windows Service via a ServiceHost. However, I've now encountered some problems debugging the Windows Service startup. This page on MSDN contains links and instructions on how to debug the startup of a Windows Service. While attempting to install the Debugging tools, I encountered a failure trying to install them as part of the Windows 7 SDK. This page on Stack Overflow had a solution, but it wasn't quite complete. This page had the missing parts.

Journey to robust Windows Services

Because I have the need for it in several projects, I'm going to be working with Windows Services implemented with .NET quite a bit in the next little while. That said, I'd like to refer to some resources for working with Windows Services:

• The MSDN page on creating windows services
• The MSDN page on debugging a Windows Service
• The MSDN page on installing and uninstalling Windows Services

Also, there are some issues that I ran into while working on my first service. I had originally planned to expose a WCF endpoint through WS-HTTP, however, I can into an unpleasant exception when I tried to do so: AddressAccessDeniedException. Apparently, the cause of this is the fact that http bindings are reserved for processes run as administrators. Fool me once, shame on me. In light of that fact, I'm switching over to using net.tcp for my local Windows Service-hosted WCF service. I'll post more here as I learn it.

Adding the Network Service account to the permissions for a file/folder/registry key etc

I'm currently developing a Windows Service that's to be run under the Network Service account (owing to the fact that it requires substantial network access). I'm just starting to learn how to program Windows Services, so this is very new to me. I just started developing a simple service to start, and when I followed the instructions on this MSDN page, I was surprised when, after trying to start my service, I got the error 005: Permission denied. I later found out that I needed to give the network service account to the folder from which the service was running, i.e. the debug output folder of my project. Doing this isn't as simple as setting the permissions for other users. To add the network service account to the permissions list in Windows 7, do the following :

1. Right click on the folder containing the service you want to debug, and go to Properties
2. Go to the Security tab, and under the box marked "Group or user names", click on Edit ...
3. Under the window that pops up, you'll see another box marked "Group or user names". Underneath that box, click on Add ...
4. In the "Select Users or Groups" dialog that pops up, click on the Advanced... button at the bottom left.
5. In the advance "Select Users or Groups" dialog that pops up, click on Find Now to find all the users, groups and built in security principals.
6. Once the search results are populated, scroll down and select "NETWORK SERVICE" (not "NETWORK") and click on Ok, then Ok again in the Select Users or Groups dialog.
7. One you're back to the Permissions window, ensure that you give the Network Service account Full Control over the folder in the Permissions box, then click Ok.
8. Click Ok one last time to close the properties for your folder, and you're done.