Solving "m_safeCertContext is an invalid handle."

I've recently been trying to get an application working in Azure App Service that acts as a client who calls out to another service via WCF with TransportWithMessageCredential mode for security and Certificate mode for authentication. I've been getting the following error:

m_safeCertContext is an invalid handle.

According to this blog post, this error gets thrown when the certificate isn't correctly imported or has incorrect trust (for any of many possible reasons). Some of those reasons can include incorrect passwords, but there are others as well, like what I was encountering: in Azure App Service, there's no local user signed on when your application is running. Because of that, you run afoul of a subtle issue with managing certificates: all of the constructors, by default, use the user certificate store to temporarily store the PrivateKey of any loaded X509Certificate2 objects. Therefore, on an Azure App Service application, unless you use the new X509Certificate2(certBytes, passwordString, X509KeyStorageFlags.MachineKeySet | X509KeyStorageFlags.Exportable) constructor with the MachineKeySet | Exportable flags, your certificate will not be correctly read and will SILENTLY FAIL!!