Troubleshooting connections to a service running in Service Fabric in Azure

Recently I decided to try deploying a Service Fabric cluster to Azure and investigate what it takes to create applications with Service Fabric. I used the default ARM template to deploy the cluster, with the parameters in the parameter file set appropriately. I've been able to successfully deploy the cluster itself, along with a private sample application that's showing as running and healthy within the cluster. (all of this with VSTS, but that's for another post). I'm now running into the problem of actually connecting to the Service.

When I use postman to connect to the service, I'm connecting to a URL like :

https://mycluster.westus.cloudapp.azure.com:8870/api/things

However, when I send my request, I instantly see Postman fail to connect:

Here's the steps taken when verifying all the settings so far:

• I got the address for my cluster from the "IP Address" resource that came with the ARM template that's in the Azure portal, using the "Copy" functionality. 
• I've verified that the Load Balancer that was set up by the ARM template is correctly configured to use my application ports.
• I've consulted the documentation for Load Balancer health probes here to ensure that my machines are using the correct type of probe: https://docs.microsoft.com/en-ca/azure/load-balancer/load-balancer-custom-probe-overview#learn-about-the-types-of-probes . In my case, I don't want to use HTTP even though I've got an HTTP service because I'd need a 200 response. Instead, we use the more basic TCP probe which determines health status based on TCP handshake, which should be just fine.
• I've updated the Diagnostics lettings on the Load Balancer to pipe logs out to a storage account. Using the generated output, I've found that my health probe contradicts my expectations and is in fact failing. What's worse, there's a timing issue: the health probe fails too many times before the Service Fabric Host can startup on the VMs and then permanently marks the hosts as failed, preventing accessibility to any of my VMs, which would seem to explain my inability to connect to my services AND the speed with which the response is returned (because the traffic doesn't even get past the Load Balancer).
• I've used Remote Desktop to gain access to the Virtual Machine Scale Set VMs thanks to the default settings that came with the Service Fabric ARM template. Loading up PowerShell and executing the command "iwr -Method Get -Uri https://localhost:8870/api/Things" yields the error "iwr : The underlying connection was closed: An unexpected error occurred on a send". It would seem that I can't even get an actual connection to my service working on the local machine. This would explain why the health probes are failing: they're perfectly legit.
• Running the command "netstat -an | ? { $_ -like '*8870*' }" on the VMSS VM indicates that the Service Fabric Host has in fact launched my process on my expected port of 8870 and the process is listening on that port. Curiously, I'm also seeing an established connection on that port as well. This is at least somewhat consistent with the fact that the Service Fabric Management Portal is showing my service as healthy on all nodes, but inconsistent with the status of the health probe.
• Figuring that I now have a past problem that I already solved, I tried setting the permissions of the certificate stores for the certificates my API application uses. After some time waiting for the load balancer health probes to update, they were now able to connect and the services were running properly.

Waiting for the network to be up on the BeagleBone Black

It finally hit me today as I was reading over this article, and it should have hit me sooner. The article mentions that in order to get past the network startup hurdles in systemd, you need to wait on the NetworkManager service. However, I use connman. It didn't occur to me until today that they provide exactly the same functionality, and I just needed to swap one with the other. Now,  I have all my network dependent programs simply After=connman.service in the systemd service descriptor files, and they're golden.

"Network is unreachable" on an embedded box

We've been working extensively with embedded boxes lately trying to migrate our platform over. One of the programs we've been trying to migrate today was a simple announcer program that sends out a UDP broadcast. On our first go, the program wouldn't work, saying that the "Network is unreachable" when doing a 'perror' after the call to 'sendto'. After much experimentation with various network binding flags, etc, which took the whole day, I went into overtime trying to figure out the problem. Eventually, I got to the point where I inspected the routing tables of the embedded system and was shocked to find .... there was no routing table entry for the default gateway, i.e. 0.0.0.0 destination IP. I'm actually kind of ashamed of how long it took me to find this, but in my defense, I was dealing with a wild mixture of systems and had done a lot of network reconfiguration with the systems and some of the stuff was handled automatically for me by my Ubuntu box, which gave me some inconsistent results at times when experimenting.  For my own future reference, here's how I solved the problem:

• Run 'route -n'
• Ensure there's an entry that looks like the following :
• Destination: 0.0.0.0 Gateway: [the desired gateway, e.g. 192.168.10.1] Genmask: 0.0.0.0 Flags: UG, Metric: 0, Ref: 0, Use: 0, Iface: eth0 (or eth2, or whatever you may have)
• If the above entry doesn't exist, add it with the following command:
• sudo route add -net 0.0.0.0 netmask 0.0.0.0 gw 192.168.10.1 dev eth0

You should now have no problems with your UDP broadcasting. If you do, well, let me know how it goes, I'd love to hear your problem and your solution.

Adding the Network Service account to the permissions for a file/folder/registry key etc

I'm currently developing a Windows Service that's to be run under the Network Service account (owing to the fact that it requires substantial network access). I'm just starting to learn how to program Windows Services, so this is very new to me. I just started developing a simple service to start, and when I followed the instructions on this MSDN page, I was surprised when, after trying to start my service, I got the error 005: Permission denied. I later found out that I needed to give the network service account to the folder from which the service was running, i.e. the debug output folder of my project. Doing this isn't as simple as setting the permissions for other users. To add the network service account to the permissions list in Windows 7, do the following :

1. Right click on the folder containing the service you want to debug, and go to Properties
2. Go to the Security tab, and under the box marked "Group or user names", click on Edit ...
3. Under the window that pops up, you'll see another box marked "Group or user names". Underneath that box, click on Add ...
4. In the "Select Users or Groups" dialog that pops up, click on the Advanced... button at the bottom left.
5. In the advance "Select Users or Groups" dialog that pops up, click on Find Now to find all the users, groups and built in security principals.
6. Once the search results are populated, scroll down and select "NETWORK SERVICE" (not "NETWORK") and click on Ok, then Ok again in the Select Users or Groups dialog.
7. One you're back to the Permissions window, ensure that you give the Network Service account Full Control over the folder in the Permissions box, then click Ok.
8. Click Ok one last time to close the properties for your folder, and you're done.