Enabling changes to Default Services within applications during deployment on a Service Fabric cluster

As it turns out, as of Service Fabric runtime 6.1, allowing applications to change their Default Services during an upgrade is not enabled by default. This can be controlled by the setting called 'EnableDefaultServicesUpgrade' in the cluster level setting group called 'ClusterManager'. This setting can be set in the Cluster Manifest if you're managing your own cluster on-premise, or it can be set in an ARM template if deploying to an Azure-based cluster like so:

{
"apiVersion": "2016-09-01",
"type": "Microsoft.ServiceFabric/clusters",
"name": "[parameters('clusterName')]",
"location": "[parameters('location')]",
"dependsOn": [
"[variables('supportLogStorageAccountName')]"
],
"properties": {
"certificate": {
"thumbprint": "[parameters('certificateThumbprint')]",
"x509StoreName": "[parameters('certificateStoreValue')]"
},
"clientCertificateCommonNames": [],
"clientCertificateThumbprints": [],
"clusterState": "Default",
"diagnosticsStorageAccountConfig": {
"blobEndpoint": "[reference(concat('Microsoft.Storage/storageAccounts/', variables('supportLogStorageAccountName')), '2017-06-01').primaryEndpoints.blob]",
"protectedAccountKeyName": "StorageAccountKey1",
"queueEndpoint": "[reference(concat('Microsoft.Storage/storageAccounts/', variables('supportLogStorageAccountName')), '2017-06-01').primaryEndpoints.queue]",
"storageAccountName": "[variables('supportLogStorageAccountName')]",
"tableEndpoint": "[reference(concat('Microsoft.Storage/storageAccounts/', variables('supportLogStorageAccountName')), '2017-06-01').primaryEndpoints.table]"
},
"fabricSettings": [
{
"parameters": [
{
"name": "ClusterProtectionLevel",
"value": "[parameters('clusterProtectionLevel')]"
}
],
"name": "Security"
},
{
"parameters": [
{
"name": "EnableDefaultServicesUpgrade",
"value": "[parameters('enableDefaultServicesUpgrade')]"
}
],
"name": "ClusterManager"
}
],
"managementEndpoint": "[concat('https://',reference(variables('lbIPName')).dnsSettings.fqdn,':',variables('nt0fabricHttpGatewayPort'))]",
"nodeTypes": [
{
"name": "[variables('vmNodeType0Name')]",
"applicationPorts": {
"endPort": "[variables('nt0applicationEndPort')]",
"startPort": "[variables('nt0applicationStartPort')]"
},
"clientConnectionEndpointPort": "[variables('nt0fabricTcpGatewayPort')]",
"durabilityLevel": "Bronze",
"ephemeralPorts": {
"endPort": "[variables('nt0ephemeralEndPort')]",
"startPort": "[variables('nt0ephemeralStartPort')]"
},
"httpGatewayEndpointPort": "[variables('nt0fabricHttpGatewayPort')]",
"isPrimary": true,
"vmInstanceCount": "[parameters('nt0InstanceCount')]"
}
],
"provisioningState": "Default",
"reliabilityLevel": "Silver",
"upgradeMode": "Automatic",
"vmImage": "Windows"
},
"tags": {
"resourceType": "Service Fabric",
"displayName": "IoT Service Fabric Cluster",
"clusterName": "[parameters('clusterName')]"
}
}

Getting Started with deploying your first Azure Service Fabric resource

I've recently started getting on the Service Fabric band wagon, running in Azure in my case. When you run Service Fabric in Azure (vs on-prem), things are a little bit different. Instead of running your own manager tool on some machine (most likely virtual), it's provided for you as a resource in the Azure Resource Manager. To get started with Service Fabric in Azure, I kicked things off by running the ARM template available in the Azure Quick starts on github.

I started off by putting the template into source control and then creating a VSTS build and automated release for the template. Before you can run the template as it is on Github (at the time of this writing) you'll need to have the following pre-requisites:

• A pre-existing key vault
• An X509 certificate populated in that key vault, stored as a .PFX file with a password to secure it.

In my case, I made up a separate resource group template to set all that up and stage it before executing the resource group template for the Service Fabric manager and it's associated VMs that would provide the nodes for the Fabric.

Once the templates were successfully executed and I had my resources created, I discovered a few more things that needed to be done before an App could be deployed to the fabric:

1. In the Virtual Machine Scale Set that gets created by the template, you have to go turn the VMSS on in the portal! ** It's not turned on by default ! **
2. As soon as the VMSS is turned on for the first time and starts connecting to the Service Fabric, it immediately starts an upgrade to the latest version of the Fabric. DO NOT TURN OFF THE VMSS DURING THIS TIME! OR YOU'LL HAVE TO START OVER. You can track the progress of the update during this initial start up using the following powershell snippet. As soon as the 'ClusterState' is 'Ready', you can start executing other operations:
1. $resourceGroupName = 'myresourcegroup'; $clusterName = 'mycluster'; Get-AzureRmServiceFabricCluster -ResourceGroupName $resourceGroupName -Name $clusterName  | select -Property Name,ClusterState,ManagementEndpoint
3. In order to connect to the Service Fabric, you need to first specify a certificate for Admin clients to connect to the Fabric. You can do this with a command similar to the following:
1. Add-AzureRmServiceFabricClientCertificate -ResourceGroupName $resourceGroupName -Name $clusterName -Thumbprint ABCDEFABCDEFABCDEFABCDEFABCDEFABCDEF3A9A -Admin
2. NOTE: DON'T specify the same certificate as both an Admin certificate and as a read-only certificate, otherwise it can confuse the cluster + browser and prevent you from being able to log in as an Administrator!
3. In order to log into the Explore via Chrome, you'll need to ensure that you explicitly import your client certificate into Chrome's certificate store, AND configure it for Client Authentication!
4. Updating the user configuration of the Fabric by doing things like adding certificates CAN TAKE AN ABSURDLY LONG TIME because adding a certificate requires publication of that certificate OUT TO EACH NODE IN THE FABRIC. And it seems like they don't do it in parallel! That's why you should shorten the timeouts associated with operations in the Fabric. You can do this via the Service Fabric Cluster resource by going to the 'Fabric upgrades' tab in the blade -> 'Advanced upgrade settings'
5. The default port in the Service Fabric Service project item in Visual Studio is http:8529. The default port for the load balancer in the Service Fabric cluster template is http:80. See a problem here ? You'll have to change one or the other to ensure they match up so that requests to your load-balancer front-end can actually get through to the machines in your cluster!
6. The very next thing you should do after sanity checking your application to make sure it's correctly configured for communications ... secure the fucking thing! Now that you've proven that you can connect via port 80 and everything's mapped correctly, disable it! Move to port 443 and secure all of your requests by default! Not a single thing should go between clients and your cluster unencrypted! Additionally, all traffic between you and your clients should, as a matter of best practice, use message-based encryption wherever possible. See this article. As a hint, you should replace the HTTP mapping rule in your Resource Group template with an HTTPS mapping rule in the load balancer.
7. Now that you've got a simple endpoint, start adding authentication and authorization and make sure you're allowing people to do only exactly that which you want them to do! TODO: include link for Service Fabric authentication and authorization!
8. If you haven't already, you should absolutely set up automated releases via VSTS (if that's your tool of choice). Ensure that you've made your Service Fabric application upgradeable by placing the following in your Cloud.xml publish profile (or whichever publish profile you may be using):
1. <UpgradeDeployment Mode="Monitored" Enabled="true">
2.     <Parameters FailureAction="Rollback" Force="True" />
3.   </UpgradeDeployment>
9. 
   

Some interesting points regarding automated deployment of Alert Rules within an Azure Resource Group template

After some recent endeavours to add Application Insights alert rules, I stumbled (quite sorely) over some quirks around defining Alert Rules within an ARM template:

1) When copying an existing definition out of the Azure portal, there will be a "hidden-link:..." prefixed tag in the tags section of the resource definition. Normally tags are simply extra metadata, but in the case of Application Insights and its related artifacts, the "hidden-link:" tag is actually a functional requirement. You can't delete it! Otherwise, the ARM template deployment will throw a very unhelpful 'Microsoft.WindowsAzure.Management.Monitoring.MonitoringServiceException' with no further details.

2) When defining Alert Rules on custom metrics, those metrics must *already* exist within the existing Application Insights instance. Otherwise, as above, the ARM template deployment will throw a very unhelpful 'Microsoft.WindowsAzure.Management.Monitoring.MonitoringServiceException' with no further details. The consequence of this is that you won't be able to define such alert rules when deploying a new resource group using the ARM template for the first time! Otherwise, it will fail and you'll be unable to execute the resource group deployment step.

Cross-compiling libexpat (and other libraries)

So, lately I've been trying to cross-compile libexpat for a new board with a different architecture from what we typically use at work, and I've been having one heck of a time, until I finally got help online from my question at stackoverflow. Turns out, I just needed to be more specific about my host architecture in the configure command:

./configure --host=arm-none-linux --enable-shared CC=arm-none-linux-gnueabi-gcc CXX=arm-none-linux-gnueabi-g++ AR=arm-none-linux-gnueabi-ar RANLIB=arm-none-linux-gnueabi-ranlib STRIP=arm-none-linux-gnueabi-strip

I'm going to try similar recompilation with other libraries so that I can improve the functionality of our older systems as well.