Troubleshooting Azure Arc for Servers – Status Offline

While trying out the capabilities of Azure Arc for Servers, I came across a couple of times when the on-premises virtual machine showed as ‘Offline' in the Azure Portal. To help this, there's a command line tool that can help reconnect the machine to Azure, called azcmagent.exe.

AzureArcOffline.jpg

1. Check the machine state

My experience as a systems administrator have ingrained in me the approach to check the simple things first. In this case, is the actually running?

If you have a way to verify connecting to or getting a response from the server another way, do that first. Many things can cause a to stop responding, including issues with the host. So we want to first check that the server is running and otherwise healthy.

2. Check the Azure Arc for Servers state
Azure Arc for Servers uses an agent known as the Connected Machine Agent. If installed successfully, you can find it listed in Add or remove programs:

AzureConnectedMachineAgent.jpg

It runs two services: the Azure Hybrid Instance Metadata Service and the Guest Configuration Service.
From a command prompt, we can use the azcmagent command to check the status of the agent:

azcmagent show

With the error below, the root cause was that one of the services was not running.
azcmagent_service.jpg

So, maybe the program was accidentally uninstalled, is being blocked by aggressive third party security software, or maybe the server has other underlying issues preventing the service from running.

Not pictured, it's also possible to get an Agent Status: Disconnected response. This indicates that things are functioning from an agent perspective, but it's not communicating with Azure. Our challenge is to re-establish that connection or figure out what's preventing it.

3. Double check the prerequisites
Azure Arc for Servers has some pre-requisites, including supported operating systems, supported Azure regions and Azure subscription and service limits. If you've previously onboarded and connected a machine successfully, make sure none of those requirements listed here, have changed.

There are no special ports required, as Azure Arc for Servers uses Port 443. If your is very restrictive, you'll want to check you're still allowed access to the required Azure URLs (preferably) or IP ranges and service tags. For details, see Networking Configuration.

4. Issue a reconnect command
If your server was previously connected and working, use the azcmagent tool to issue a reconnect command. This is useful if the server has been turned off (for more than 45 days) and its local has expired, but the machine still appears in Azure. It's also a useful step if the agent status in the Azure Portal is reporting as Offline. This step will require you to provide valid Azure credentials, in the form of a service principal, an access token or via the Azure login screen.

azcmagent reconnect --tenant-id 

This interactive login method prompts you to visit https://aka.ms/devicelogin in a browser, log in with your credentials and enter the unique code displayed in the command response. Note: by default, Internet Explorer runs in Enhanced Protected Mode on a , which blocks the functionality of this website. To enable it, use Server Manager to bring up the Local Server properties and click IE Enhanced Security Configuration: On, to turn that protection off. Make sure you re-enable it again once you have successfully logged in.

IE_Security.jpg

The preferred method is to use a service principal, which requires you to provide:

azcmagent reconnect --service-principal-id  --service-principal-secret  --tenant-id 

If you didn't generate a service principal when you first registered the machine, you can still do so by following these steps – Create a Service Principal for onboarding at scale.

Then hopefully, our azcmagent show command should get a Connected response, similar to this:
azcmagent_connected.jpg

5. Disconnect and Connect
Another use of the azcmagent tool is to remove a server if the Azure Arc Connected Machine resource has been deleted from Azure (especially accidentally), and then to connect it again, creating a new resource. This does not remove or reinstall the agent and it's services on the virtual machine. Both parameters do require you to provide valid credentials, in the way the disconnect method does.

azcmagent disconnect --service-principal-id  --service-principal-secret  --tenant-id 
azcmagent connect --service-principal-id  --service-principal-secret  --tenant-id  --subscription-id  --resource-group  --location 

Learn more:
For more information, visit Managing and maintaining the Connected Machine agent.

-SCuffy

 

This article was originally published by Microsoft’s Networking Blog. You can find the original article here.