Step By Step Troubleshooting Azure Arc-enabled servers with agent connection issues #Windows #WindowsServer #WinServ #Azure #AzureArc #Cloud

Azure Arc-enabled servers enables you to manage your Windows and physical servers and hosted outside of Azure, on your corporate , or other cloud provider. This management experience is designed to be consistent with how you manage native Azure . When a hybrid machine is connected to Azure, it becomes a connected machine and is treated as a resource in Azure. Each connected machine has a Resource ID enabling the machine to be included in a resource group. Now you can benefit from standard Azure constructs, such as Azure Policy and applying tags.

When running Azure Arc for some time and suddenly the response stopped you need to dig a bit deeper into the how things are working instead of just kicking off an MSI and the issue is still not fixed.

This is all test So it may look different in your site.just to say so.

Here I have my two servers managed by Arc

Azure Arc-enabled server

As you can see “Something went wrong while getting your resources. Please try again later.”

Azure Arc-enabled server

yes let me get more info about this as currently I know nothing about the error.

image

Azure Arc-enabled server

So It is all OK according to the Azure troubleshooter and still it doesn't work

Let me click around and see if there is and error ( I could see the local event log of the server but that's no fun Who uses this ? post some comments in the blog post) Eventlogs are extremely helpful on finding issues or hidden issue's Often people for get to look at his and see the problem right there. and yes it needs to be fixed also. 

image

Will that be the issue ?  checking already running the latest version, so what is this error or did it go wrong when updating the agent, well I did skip patching for some time on these servers and upgraded these to 2022

Let me check the agent version,  well the latest version for now..

image

How is this Azure arc be configured anyway, there is no console other than in azure and an MSI with an agent,

let me check the configuration of this and see if I can find something there.

C:ProgramDataGuestConfig

imageimage

Perfect lots of log files and a config let me check this all

image

time=”2021-09-01T16:32:17+02:00″ level=error msg=”Could not acquire token from cert: FromAssertion(): http call(https://login.windows.net/-d391a79950b1/oauth2/v2.0/token)(POST) error: reply status code was 401:n{“error”:”invalid_client”,”error_description”:”AADSTS700027: Client assertion contains an invalid signature. [Reason – The key used is expired., Thumbprint of key used by client: ‘C2FA453DD43C16E584868C1C762DC91EBEC63232', Found key ‘Start=11/12/2019 15:45:00, End=02/10/2020 15:45:00', Please visit the Azure Portal, Graph Explorer or directly use MS Graph to see configured keys for app Id ‘a16df9d0-f012-45ae-8a92-1d0ad72e045e'. Review the documentation at https://docs.microsoft.com/en-us/graph/deployments to determine the corresponding service endpoint and https://docs.microsoft.com/en-us/graph/api/application-get?view=graph-rest-1.0&tabs=http to build a query request URL, such as ‘https://graph.microsoft.com/beta/applications/].rnTrace ID: 932-2ba8-4098-813e-05a2900rnCorrelation ID: 66a070fe-6ae4-4a25-ad3f-rnTimestamp: 2021-09-01 14:32:07Z”,”error_codes”:[700027],”timestamp”:”2021-09-01 14:32:07Z”,”trace_id”:”932e7194-2ba8-4098-813e-343df05a2900″,”correlation_id”:”-4a25-ad3f-160f98c9fd9e”,”error_uri”:”https://login.windows.net/error?code=700027“}”

Seeing the Config and also see the issue here — Client assertion contains an invalid signature. [Reason – The key used is expired–

As I did not update the agent the got expired make sense.

But the device has already the new agent So reconnect ? but how ?

Looking at the Config I see all the details how the agent is been registered and the resource group etc

C:ProgramDataAzureConnectedMachineAgentConfig

agentconfig.json

{“subscriptionId”:”f34″,”resourceGroup”:”AzureBackupRG_westeurope_1″,”resourceName”:”Hyperv1201″,”tenantId”:”0b1″,”location”:”westus2″,”vmId”:”9659193c-f4d8-4a77-b8f9baad507ce9a9″,”certificateThumbprint”:”c2fa453dd43c16e584868c1c762dc91ebec63232″,”clientId”:”0-f012-45ae-8a92-1045e”}

Let me open powershell and maybe I got more details. and reactivate the Agent

With the azcmagent command you can get more details.

image

let me get all the logs

azcmagent logs

image

now we have all the logs in a zip file this could be handy for a next time.

Azure Arc-enabled server

As I reconfigure the agent with the following command

& “$env:ProgramFilesAzureConnectedMachineAgentazcmagent.exe” connect –resource-group “AzureBackupRG_westeurope_1” –tenant-id “your tenant id” –location “westus2” –subscription-id “errryh934” –verbose

With the reconnect we need to log in again and all goes well

imageimage

But in the logging there is suddenly another error

image

When looking here I see there is an Azure Policy that demands a TAG and this is currently not available on the resource group So I Can't onboard my Azure Arc server.

Thought this was about an Agent that has an expired .

Azure Arc-enabled server

Seems there is a Azure policy that is blocking as the hyperv1201 has no tags set the mvpdc02 has only a tag set.

image

image

image

After a quick change I rerun the command line and it worked perfectly and it showed up in the console again.

& “$env:ProgramFilesAzureConnectedMachineAgentazcmagent.exe” connect –resource-group “MVPRSG-Azure-Arc” –tenant-id “3078684f-d143-440a-ae40-d391a79950b1” –location “West US 2” –subscription-id “df1e2f32-7adf-48f6-b969-f02376152934” –verbose

image

Starting client connection on: \.pipehimds”
time=”2021-09-01T17:12:53+02:00″ level=debug msg=”Awaiting status message from agent…”
time=”2021-09-01T17:12:53+02:00″ level=debug msg=”Status Message received”

image

As I have a second machine with the same issue I removed the machine directly in the arc portal and rerun the registration as the agent was also already installed. (this would be the quick fix for this)

Azure Arc-enabled server

Perfect reconnecting and waiting for the Agent.

Azure Arc-enabled server

Now I can look at the Azure Arc Insights again.

Flickr Tags: Windows Server 2022,CloudOS

Follow Me on Twitter @ClusterMVP

Follow My blog https://robertsmit.wordpress.com

Linkedin Profile Http://nl.linkedin.com/in/robertsmit

Author: Robert Smit [MVP]

Robert Smit is Senior Technical Evangelist and is a current Microsoft MVP in Clustering as of 2009.
Robert has over 20 years experience in IT with experience in the educational, health-care and finance industries.
Robert's past IT experience in the trenches of IT gives him the knowledge and insight that allows him to communicate effectively with IT professionals
who are trying to address real concerns around business continuity, disaster recovery and regulatory compliance issues. Robert holds the following certifications:
MCT – Microsoft Certified Trainer, MCTS – , MCSE, MCSA and MCPS. He is an active participant in the Microsoft newsgroup community and is currently focused on Hyper-V, Clustering, SQL Server, Azure and all things related to Cloud Computing and Infrastructure Optimalization.
Follow Robert on Twitter @ClusterMVP
Or follow his blog https://robertsmit.wordpress.com
Linkedin Profile Http://nl.linkedin.com/in/robertsmit

Robert is also capable of transferring his knowledge to others which is a rare feature in the field of IT. He makes a point of not only solving issues but also of giving on the job training of his colleagues.

A customer says ” Robert has been a big influence on our technical staff and I have to come to know him as a brilliant specialist concerning Microsoft Products. He was Capable with his in-depth knowledge of Microsoft products to troubleshoot problems and develop our infrastructure to a higher level. I would certainly hire him again in the future. ”

Details of the Recommendation: “I have been coordinating with Robert implementing a very complex system. Although he was primarily a Microsoft infrastructure specialist; he was able to understand and debug .Net based complext Windows applications and websites. His input to improve performance of applications proved very helpful for the success of our project

 

This article was originally published by The Windows Server HA Blog. You can find the original article here.