Companies with large branch offices often must make a tradeoff between user experience and security. To increase employee productivity, it may make sense to deploy replicas of certain applications like Active Directory Domain Controllers or file servers in a branch office. But with limited — if any — IT resources at the remote location, how can you ensure your data is secure?
This is where shielded virtual machines come to the rescue! Shielded VMs allow you to encrypt the data of virtual machines to prevent someone with access to the Hyper-V server from accessing your sensitive data. To ensure the Hyper-V host is trusted to run your VM, it needs to prove to the Host Guardian Service (HGS) that it is configured as-expected and retrieve the keys needed to start up your shielded VM. Typically, HGS would reside in your primary datacenter where it can be better protected and managed by your security team.
With Windows Server, version 1709, we’ve made it easier to support branch offices with poor WAN links back to the main datacenter. This is done by installing a virtualized HGS server in the branch office. To run a local copy of HGS in a branch office, you would want to shield it to protect the key material from being stolen. But how do you bootstrap the first shielded HGS server? It needs another HGS to release its keys. That’s why we’ve introduced fallback URLs in Hyper-V to allow your Hyper-V server to prefer a local HGS server and fall back to your main datacenter’s HGS in case the local server is offline.
This blog post explains the recommended architecture for setting up HGS in a branch office scenario.
To set up a branch office HGS, follow these high-level steps:
- Ensure your main datacenter’s HGS cluster is installed and operational. For more information, read our HGS deployment instructions.
- Ensure your branch office has at least one Hyper-V host running Windows Server, version 1709 or higher and network connectivity to your main datacenter. Register your branch office Hyper-V host with your main datacenter’s HGS cluster by following our guarded host deployment instructions. At this point, since you only have one HGS cluster, the Hyper-V host should use the datacenter’s HGS server as the primary HGS server.
- Deploy a new shielded VM running Windows Server 2016 on your branch office Hyper-V server. This will become your branch office HGS server. Note that while the Hyper-V host must run Windows Server, version 1709, we recommend using Windows Server 2016 for the HGS servers.
- Configure the new shielded VM as an HGS server following our HGS deployment instructions. This HGS server should not join the existing HGS cluster, nor should it use the same certificates as the main datacenter’s HGS servers. You can, however, join the new HGS server to the same Active Directory domain, if desired. You can optionally add more HGS servers in the branch office for resiliency, but since you have the main datacenter as a fallback option, you may only need one HGS server running in the remote office.
- Register the Hyper-V host (and any others in your branch office) with your new HGS server. Be sure to keep both the branch office HGS and main datacenter HGS servers updated with your latest Hyper-V host attestation information. Your operations team should ensure both servers get updated whenever a Hyper-V host configuration changes in the branch office.
- Reconfigure the Hyper-V host to use the new branch office HGS server as its primary HGS server and the main datacenter’s HGS server as a fallback with Set-HgsClientConfiguration. Details on how to do this are included at the end of the blog post.
- Finally, deploy the rest of your shielded VMs in the branch office. It is recommended that you configure those VMs with the guardians of both the branch office and main datacenter HGS servers so that either HGS can start them up.
You can repeat these steps for each branch office you want to configure. In general, we have three core recommendations for branch office deployments:
- Every branch should use a different HGS cluster with unique certificates. Don’t join the branch office HGS nodes to the primary datacenter’s HGS cluster.
- Keep the attestation policies in sync between the primary datacenter and branch office HGS nodes. If the branch office HGS goes down, your Hyper-V host will need to pass attestation with your main datacenter’s HGS.
- For the best resiliency, configure every shielded VM at the branch office to authorize both the branch office and main datacenter HGS guardians to decrypt the VM. The branch office HGS servers must always allow the main datacenter HGS to start them up. For example, the VMs in your branch office may be configured such that:
- The shielded HGS server is configured with an owner guardian (central IT) and one additional guardian for the main datacenter HGS
- An application server shielded VM is configured with an owner guardian (business group IT) and two additional guardians, one for the shielded HGS server running in the branch office and one for the main datacenter in case the branch office HGS is offline.
Configuring fallback URLs in Hyper-V
To use the new fallback functionality, you will need to use Hyper-V on Windows Server, version 1709. Read more about the semi-annual channel to decide if upgrading to version 1709 is right for you.
You will need the URLs for both your primary (typically, the branch office HGS) and fallback (main datacenter) servers to continue. Once you have them, run the following command in PowerShell on the Hyper-V host to configure the URLs:
(Don’t forget to swap “primaryhgs” and “fallbackhgs” for your own DNS names!)
Set-HgsClientConfiguration -KeyProtectionServerUrl http://primaryhgs/KeyProtection -AttestationServerUrl http://primaryhgs/Attestation -FallbackKeyProtectionServerUrl http://fallbackhgs/KeyProtection -FallbackAttestationServerUrl http://fallbackhgs/Attestation
You can check if your Hyper-V host is passing attestation by using Test-HgsClientConfiguration:
Test-HgsClientConfiguration -UsePrimary Test-HgsClientConfiguration -UseFallback
If you are no longer using a fallback HGS server, you can remove it by using Set-HgsClientConfiguration without fallback URLs:
Set-HgsClientConfiguration -KeyProtectionServerUrl http://primaryhgs/KeyProtection -AttestationServerUrl http://primaryhgs/Attestation
When is a fallback server used?
Hyper-V will only use a fallback HGS server if it is configured with fallback URLs and the primary HGS server is unreachable or returns another non-transient error. A common example of this would be if the primary HGS servers were offline, resulting in the host being unable to establish a connection. The Hyper-V host will try to contact the primary HGS server a second time before moving to the fallback URL after a maximum of 120 seconds without a response. It will always try to use the primary URL first, even if it just used the fallback URL for its last attestation attempt or key unwrapping. If Hyper-V can reach the primary HGS server, but fails attestation or receives another terminating response, it will not attempt the fallback URL.
You can monitor the Microsoft-Windows-HostGuardianService-Client/Operational event log for event ID 2021 on a Hyper-V host to see when a fallback server is in use.
We hope you’ll find this fallback feature helpful when you’re trying to secure your branch office IT. As always, if you have any questions, reach out to us at firstname.lastname@example.org.