Getting the ETW messages for a live running instance of a Service Fabric application

As our company has recently discovered, it would be very handy to have a live running stream of diagnostic from a live running Service Fabric application when one is trying to figure out why a Service won't start.

This docs.microsoft.com article provides a handy way to enable the log streaming feature in Visual Studio for a live running service in the same manner as one would get when debugging the Service Fabric Application in your local cluster.

Using Ninject in a Self-Hosted environment, e.g. a Windows Service

Windows Services are a great way to host WCF, typically in scenarios where on-premise services are a requirement. As Dependency Injection becomes a standard practice in large scale systems, it can be hard to choose the right DI container. One of my personal favourites in Ninject because of its (typical) ease of configuration. Some benchmarks will show that Ninject isn't the fastest container out there in terms of constructing objects, but in practical use I've never once found that to be a problem.

The following example shows a function that can be used in a Windows Service for setting up Ninject to create ServiceHost instances in a self-hosting scenario:

private static NinjectServiceHost CreateServiceHost(IKernel standardKernel, Type serviceType)
{
 if (standardKernel == null)
 {
  throw new ArgumentNullException("standardKernel");
 }
 
 if (serviceType == null)
 {
  throw new ArgumentNullException("serviceType");
 }
 
 NinjectServiceHost ninjectServiceHost = new NinjectServiceHost(
  serviceBehavior: new NinjectServiceBehavior(
   instanceProviderFactory: type => new NinjectInstanceProvider(type, standardKernel),
   requestScopeCleanUp: new WcfRequestScopeCleanup(true)
   ),
  serviceType: serviceType
 );
 
 return ninjectServiceHost;
}

This function will be good enough to boot strap your services in your Windows ServiceBase

implementation in the vast majority of cases. You need only pass in your pre-configured 

IKernel instance and the Type of the WCF Service implementation.

Unit testing Service Fabric Reliable Service and Actors with Mocks and Moqs

Install-Package Moq
Install-Package ServiceFabric.Mocks

Some key bits of code you'll need:

this.mockRepository = new MockRepository(MockBehavior.Strict);

var statefulServiceContext = MockStatefulServiceContextFactory.Create(
codePackageActivationContext: this.mockRepository.Create<ICodePackageActivationContext>().Object,
serviceTypeName: String.Concat(typeof(MyActorService).Name, "Type"),
serviceName: new Uri("fabric:/MockApplicationType/MyActorServiceType"),
partitionId: Guid.NewGuid(),
replicaId: 1
);

this.mockServiceDependencyThing = this.mockRepository.Create<IServiceDependencyThing>();

MockActorService<MyActorService> actorServiceForActor = MockActorServiceFactory.CreateActorServiceForActor<MyActorService>((actorService, actorId) => new MyActorService(actorService, actorId, this.mockServiceDependencyThing.Object));

this.sut = new MyActorService(actorServiceForActor, new ActorId(Guid.NewGuid()), this.mockServiceDependencyThing.Object);

Adding Dependency Injection support for Service Fabric Stateful Services and Actors

Check this page out: https://alexmg.com/introducing-the-autofac-integration-for-service-fabric/

Right from the maker of Autofac. You'll need to import the Autofac.ServiceFabric NuGet package, which is currently in beta.

Note: As of the current version (1.0.0-beta1) the InternalsVisibleTo functionality mentioned in the article doesn't work, so you ** HAVE TO ** set your registered Actors and Reliable Services to "public" visibility.

Note 2: The dependencies on the Autofac.ServiceFabric NuGet package aren't set correctly, so you'll have to "Update-Package Autofac -Version 4.6.0" since 4.5.0 isn't good enough.

Adding Dependency Injection support for Service Fabric Stateless API applications

Turns out that it's pretty easy (for Stateless Services anyway).

In your Package Manager Console:

Install-Package Autofac.WebApi2 -Version 4.0.1

In your Startup.cs file (assuming you're using the Visual Studio Project templates):

Startup.cs:

public static void ConfigureApp(IAppBuilder appBuilder)
{
// Configure Web API for self-host.
HttpConfiguration config = new HttpConfiguration();

config.Routes.MapHttpRoute(
name: "DefaultApi",
routeTemplate: "api/{controller}/{id}",
defaults: new { id = RouteParameter.Optional }
);

var containerBuilder = new ContainerBuilder();

containerBuilder.RegisterModule<MyApiModule>();

config.DependencyResolver = new AutofacWebApiDependencyResolver(containerBuilder.Build());

appBuilder.UseWebApi(config);
}

And, assuming you've built an Autofac Module, it should look something vaguely like this:

public class PlayDatesApiModule : Autofac.Module

{

protected override void Load(ContainerBuilder builder)

{

base.Load(builder);

builder.RegisterType<MyDependencyConcreteType>().As<IMyDependency>();

builder.RegisterType<ValuesController>();

}

}

Adding Azure Active Directory OAuth 2.0 authentication to a Service Fabric Web API (Stateless) service

... is pretty much the same as adding it to a normal Web API 2.0 application:

[Authorize]
public class ValuesController : ApiController
{

        }

Then in your Startup.cs file:

// This code configures Web API. The Startup class is specified as a type

// parameter in the WebApp.Start method.

public static void ConfigureApp(IAppBuilder appBuilder)

{

CodePackageActivationContext activationContext = FabricRuntime.GetActivationContext();

ConfigurationPackage configurationPackageObject = activationContext.GetConfigurationPackageObject("Config");

ConfigurationSection configurationSection = configurationPackageObject.Settings.Sections["ActiveDirectoryServiceConfigSection"];

appBuilder.UseWindowsAzureActiveDirectoryBearerAuthentication(

new WindowsAzureActiveDirectoryBearerAuthenticationOptions

{

Tenant = configurationSection.Parameters["TenantName"].Value,

TokenValidationParameters = new TokenValidationParameters

{

ValidAudience = configurationSection.Parameters["AppIdUri"].Value

},

Provider = new OAuthBearerAuthenticationProvider

{

OnValidateIdentity = OnValidateUserIdentityAsync

}

});

// Configure Web API for self-host. 

HttpConfiguration config = new HttpConfiguration();

config.Routes.MapHttpRoute(

name: "DefaultApi",

routeTemplate: "api/{controller}/{id}",

defaults: new { id = RouteParameter.Optional }

);

appBuilder.UseWebApi(config);

}

The trick here is to ** ENSURE THAT WAAD BEARER AUTHENTICATION GETS REGISTERED BEFORE REGISTERING WEB API!!! **

Getting Started with deploying your first Azure Service Fabric resource

I've recently started getting on the Service Fabric band wagon, running in Azure in my case. When you run Service Fabric in Azure (vs on-prem), things are a little bit different. Instead of running your own manager tool on some machine (most likely virtual), it's provided for you as a resource in the Azure Resource Manager. To get started with Service Fabric in Azure, I kicked things off by running the ARM template available in the Azure Quick starts on github.

I started off by putting the template into source control and then creating a VSTS build and automated release for the template. Before you can run the template as it is on Github (at the time of this writing) you'll need to have the following pre-requisites:

• A pre-existing key vault
• An X509 certificate populated in that key vault, stored as a .PFX file with a password to secure it.

In my case, I made up a separate resource group template to set all that up and stage it before executing the resource group template for the Service Fabric manager and it's associated VMs that would provide the nodes for the Fabric.

Once the templates were successfully executed and I had my resources created, I discovered a few more things that needed to be done before an App could be deployed to the fabric:

1. In the Virtual Machine Scale Set that gets created by the template, you have to go turn the VMSS on in the portal! ** It's not turned on by default ! **
2. As soon as the VMSS is turned on for the first time and starts connecting to the Service Fabric, it immediately starts an upgrade to the latest version of the Fabric. DO NOT TURN OFF THE VMSS DURING THIS TIME! OR YOU'LL HAVE TO START OVER. You can track the progress of the update during this initial start up using the following powershell snippet. As soon as the 'ClusterState' is 'Ready', you can start executing other operations:
1. $resourceGroupName = 'myresourcegroup'; $clusterName = 'mycluster'; Get-AzureRmServiceFabricCluster -ResourceGroupName $resourceGroupName -Name $clusterName  | select -Property Name,ClusterState,ManagementEndpoint
3. In order to connect to the Service Fabric, you need to first specify a certificate for Admin clients to connect to the Fabric. You can do this with a command similar to the following:
1. Add-AzureRmServiceFabricClientCertificate -ResourceGroupName $resourceGroupName -Name $clusterName -Thumbprint ABCDEFABCDEFABCDEFABCDEFABCDEFABCDEF3A9A -Admin
2. NOTE: DON'T specify the same certificate as both an Admin certificate and as a read-only certificate, otherwise it can confuse the cluster + browser and prevent you from being able to log in as an Administrator!
3. In order to log into the Explore via Chrome, you'll need to ensure that you explicitly import your client certificate into Chrome's certificate store, AND configure it for Client Authentication!
4. Updating the user configuration of the Fabric by doing things like adding certificates CAN TAKE AN ABSURDLY LONG TIME because adding a certificate requires publication of that certificate OUT TO EACH NODE IN THE FABRIC. And it seems like they don't do it in parallel! That's why you should shorten the timeouts associated with operations in the Fabric. You can do this via the Service Fabric Cluster resource by going to the 'Fabric upgrades' tab in the blade -> 'Advanced upgrade settings'
5. The default port in the Service Fabric Service project item in Visual Studio is http:8529. The default port for the load balancer in the Service Fabric cluster template is http:80. See a problem here ? You'll have to change one or the other to ensure they match up so that requests to your load-balancer front-end can actually get through to the machines in your cluster!
6. The very next thing you should do after sanity checking your application to make sure it's correctly configured for communications ... secure the fucking thing! Now that you've proven that you can connect via port 80 and everything's mapped correctly, disable it! Move to port 443 and secure all of your requests by default! Not a single thing should go between clients and your cluster unencrypted! Additionally, all traffic between you and your clients should, as a matter of best practice, use message-based encryption wherever possible. See this article. As a hint, you should replace the HTTP mapping rule in your Resource Group template with an HTTPS mapping rule in the load balancer.
7. Now that you've got a simple endpoint, start adding authentication and authorization and make sure you're allowing people to do only exactly that which you want them to do! TODO: include link for Service Fabric authentication and authorization!
8. If you haven't already, you should absolutely set up automated releases via VSTS (if that's your tool of choice). Ensure that you've made your Service Fabric application upgradeable by placing the following in your Cloud.xml publish profile (or whichever publish profile you may be using):
1. <UpgradeDeployment Mode="Monitored" Enabled="true">
2.     <Parameters FailureAction="Rollback" Force="True" />
3.   </UpgradeDeployment>
9. 
   

Solving "System.Net.WebException: The remote server returned an error: (417) Expectation Failed" with a WCF service

I recently started getting the following error message when trying to connect to a web service we had put on an Azure VM running behind an Azure load balancer:

System.ServiceModel.ProtocolException: The remote server returned an unexpected response: (417) Expectation Failed. ---> System.Net.WebException: The remote server returned an error: (417) Expectation Failed.

It turns out the fix was to put the following element in my <configuration> :

<system.net>
<settings>
<!-- This is required when running in Azure VMs behind an Azure load balancer -->
<servicePointManager expect100Continue="true" />
</settings>
</system.net>

Solving "System.Net.WebException: The remote server returned an error: (417) Expectation Failed" with a WCF service

I recently started getting the following error message when trying to connect to a web service we had put on an Azure VM running behind an Azure load balancer:

System.ServiceModel.ProtocolException: The remote server returned an unexpected response: (417) Expectation Failed. ---> System.Net.WebException: The remote server returned an error: (417) Expectation Failed.

It turns out the fix was to put the following element in my <configuration> :

<system.net>
<settings>
<!-- This is required when running in Azure VMs behind an Azure load balancer -->
<servicePointManager expect100Continue="true" />
</settings>
</system.net>

Solving "Binding validation failed because the binding's MsmqAuthenticationMode property is set to WindowsDomain but MSMQ is installed with Active Directory integration disabled"

I've been trying to set up remote reads from an MSMQ queue to a WCF service hosted on another machine. I seem to have all my application settings correct, but I'm being screwed over by machine-level configuration on the service machine: the service machine won't run in Active Directory mode, no matter what I do.

After searching for the problem, I found this page on John Breakwell's MSDN blog. It describes the problem: essentially there's a legacy msmq object left around when you install MSMQ Server and Active Directory Integration features together in the same transaction. Here's how you **actually** fix the problem:

1. Uninstall all of the MSMQ features.
2. Execute this PowerShell command (or one similar to it) to actually delete the Active Directory object described in the article (which doesn't go into nearly enough detail on this point):  get-adobject -filter “name -eq ‘msmq'” | Where { $_.DistinguishedName -eq ‘CN=msmq,CN=MyAffectedServerNameHere,OU=Web Servers,OU=Member Servers,DC=MyDomainNameHere,DC=Network,DC=ads’} | Remove-ADObject
3. Reinstall **just** MSMQ server, then reboot your machine.
4. Reinstall **just** MSMQ active directory integration, then reboot your machine.

You should now be good to go.

Solving "Binding validation failed because the binding's MsmqAuthenticationMode property is set to WindowsDomain but MSMQ is installed with Active Directory integration disabled"

I've been trying to set up remote reads from an MSMQ queue to a WCF service hosted on another machine. I seem to have all my application settings correct, but I'm being screwed over by machine-level configuration on the service machine: the service machine won't run in Active Directory mode, no matter what I do.

After searching for the problem, I found this page on John Breakwell's MSDN blog. It describes the problem: essentially there's a legacy msmq object left around when you install MSMQ Server and Active Directory Integration features together in the same transaction. Here's how you **actually** fix the problem:

1. Uninstall all of the MSMQ features.
2. Execute this PowerShell command (or one similar to it) to actually delete the Active Directory object described in the article (which doesn't go into nearly enough detail on this point):  get-adobject -filter “name -eq ‘msmq'” | Where { $_.DistinguishedName -eq ‘CN=msmq,CN=MyAffectedServerNameHere,OU=Web Servers,OU=Member Servers,DC=MyDomainNameHere,DC=Network,DC=ads’} | Remove-ADObject
3. Reinstall **just** MSMQ server, then reboot your machine.
4. Reinstall **just** MSMQ active directory integration, then reboot your machine.

You should now be good to go.

Unable to add multiple listen rule keys or subscriptions to a Topic in an Azure Service Bus in an Azure Resource Group template

I recently ran across a problem wherein I was unable to add multiple Listen rules for my applications to my Topics in Azure Service Bus within an Azure Resource Group Template. I would run my template and it would fail to create the keys for the Topic or the Subscription for the topic of there was more than one of either of those for the Topic. It wouldn't even be consistent: it would switch back and forth between them between runs of the template. After a discussion with one of the Solution Architects at Microsoft, I found out there is, at the time of this writing, a bug in the Azure Resource Manager for Topics within Service Buses that prevents Topic access rules and subscriptions from being simultaneously created. As a work around, I was instructed to introduce an artificial dependency between each of the subscriptions to force the resource manager to create them serially rather than in parallel and this did the trick.

For example:

// TODO:

Avoiding Windows Service startup timeouts in your code

check this out: https://www.devopsonwindows.com/avoid-windows-service-start-up-timeouts/

Finally ... how to debug the startup of a Windows Service application

Do what this guy says: http://einaregilsson.com/run-windows-service-as-a-console-program/

Pasted here for posterity, here's the example of how to write a Windows Service such that it can execute in a console and a developer can debug the startup routine:

using System;

using System.ServiceProcess;

 

public partial class DemoService : ServiceBase

{

    static void Main(string[] args)

    {

        DemoService service = new DemoService();

 

        if (Environment.UserInteractive)

        {

            service.OnStart(args);

            Console.WriteLine("Press any key to stop program");

            Console.Read();

            service.OnStop();

        }

        else

        {

            ServiceBase.Run(service);

        }

 

    }

    public DemoService()

    {

        InitializeComponent();

    }

 

    protected override void OnStart(string[] args)

    {

        // TODO: Add code here to start your service.

    }

 

    protected override void OnStop()

    {

        // TODO: Add code here to perform any tear-down

        //necessary to stop your service.

    }

}