Troubleshooting Windows 10 Update for Business by Using Azure Update Compliance and Log Analytics

Cory Roberts and Tan Tran

Dear IT Pros,

Recently I and Cory Roberts, Microsoft Sr. CE, worked together on a Customer's Project, We were upgrading roughly eight thousands Windows 10 devices from multiple versions of Windows 10, (1803, 1809, 1903, 1909) to the current branch 2004. The upgrade deployment has been proceeded with Microsoft Endpoint Manager.

In Endpoint Manager, besides Device Status and End User Update Status, there was not much data provided by Endpoint Monitor or Log. It was hard to  the windows 10 feature update process… We decided to go with Azure Update Compliance and Azure Log Analytics Query for monitoring and  the Windows Feature Update deployment to match our Customer's need.

The steps to use Log Analytics for troubleshooting of Endpoint Manager Deployment on Windows 10 Feature Update as follow:

  1. In Endpoint Manager, create Windows 10 Feature Update Deployment and assign to the related Device Group.
  2. Create Log Analytics Workspace (if you do not have one).
  3. Install Update Compliance from Azure Market Place
  4. Onboarding Update Compliance for Windows 10 devices
  5. Set Windows 10 Clients to forward telemetry data to Log Analytics Workspace.
  6. Using Kusto Queries to Monitor and the Upgrade Process.

In Endpoint Manager, create Windows 10 Feature Update Deployment and assign to the related Device Group

– In Endpoint ManagerDevices,

– Windows 10 Feature Update, Create Profile

tantran_1-1599647128699.png

– Choose the update to deploy

tantran_2-1599647128701.png

– Assign to Device Group and create the deployment.

Create Log Analytics Workspace (if you do not have one).

  • In Azure Portal, search for log analytic workspace

tantran_3-1599647128703.png

  • Creating the Log Analytic Workspace:

tantran_5-1599647128625.png

  • Configure Resource Group and location for Log Analytics Workspace

tantran_6-1599647128628.png

  • Click Create

tantran_7-1599647128629.png

Install Azure Update Compliance from Market Place:

Update Compliance uses Windows 10 diagnostic data for all of its reporting. It collects system data including update deployment progress, Windows Update for Business configuration data, and Delivery Optimization usage data, and then sends this data to a customer-owned Azure Log Analytics workspace to power the experience.

  • Update Compliance works only with desktops of Windows 10 Professional, Education, and Enterprise editions. It is not support for Windows Server, Surface Hub, IoT.
  • Update Compliance required windows 10 device telemetry at minimum basic level and a Commercial ID, a globally-unique identifier assigned to a specific Solution of Log Analytics workspace.
  • After Update Compliance is configured, it could take 48-72 hours before they first appear and continue refreshing its data every 12 hour
  • Update Compliance also provide Windows Update Delivery Optimization Status (WUDOAggregratedStatus, WUDOStatus), and Windows Threat and Update status (WDAV Threat, WDAVStatus)

To Install Azure Update Compliance

  • Go to Azure search and type Update Compliance,
  • Choose MarketplaceUpdate Compliance

tantran_8-1599647128723.png

  • Choose the same LogAnalytics Workspace

tantran_9-1599647128633.png

  • Create

Now, the Update Compliance Log will be available for Query search in Log Analytics Workspace as shown here:

tantran_13-1599647128704.png

 To Configure GPO for Update Compliance Clients

  • Go to Computer Configuration>Administrative Templates>Windows ComponentsData Collection and Preview Build
  • Choose “Allow Telemetry” and set level of diagnostic to at least basic level
  • Choose “Configure the Commercial ID” and copy and paste ID from WaaSUpdateInsight to the  setting box

You could view the Commercial ID from the WaaSUpdateInsight as shown:

tantran_14-1599647128639.png

  • Choose “Allow device name to be sent in Windows diagnostic data” and Enabled

Onboarding Update Compliance for Windows 10 Devices.

The Update Compliance Configuration Script is the recommended method of configuring devices to send Telemetry data to Azure Log Analytics Workspace for use with Update Compliance. The script configures device policies via Group Policy, ensures that required services are running, and more.

You can download the script here.

The script is organized into two folders Pilot and Deployment. Both folders have the same key files: ConfigScript.ps1 and RunConfig.bat.

You configure RunConfig.bat according to the directions in the .bat itself, which will then execute ConfigScript.ps1 with the parameters entered to RunConfig.bat.

  • The Pilot folder is more verbose and is intended to be use on an initial set of devices and for troubleshooting. Pilot script will collect and output detailed logs
  • The Deployment folder is intended to be deployed across an entire device population in a specific environment once devices in that environment have been validated with the Pilot script.
  • Configure commercialIDValue in RunConfig.bat to your CommercialID.
  • Use a management tool like Configuration Manager or Intune to broadly deploy the script to your entire target population.

Steps to Deploy Update Compliance to Clients:

  • Edit PilotRunConfig.bat with Commercial ID of your WaaSInsight Solution and Location for log folder.
  • Run the PilotRunConfig.bat and generate errors in report from log folder
  • Review log files and correct the problems.
  • Edit DeploymentRunConfig.bat with Commercial ID of your WaaSInsight Solution and Location for log folder.
  • Run the DeploymentRunConfig.bat. Process may take more than 48hrs for collected data to show up in the Update Compliance Dashboard

tantran_15-1599647128706.png

Set Windows 10 Clients Agent to forward data to Log Analytics Workspace.

Deploy Microsoft Monitoring Agents (MMA) as installation application to all Windows 10 Clients using SCCM.

  • Download MMASetup-AMD64.exe and use 7-zip to extract MOMAgent.msi from MMASetup-AMD64.exe
  • Create SCCM MMA application using the following command:

msiexec /i MOMAgent.msi ADD_OPINSIGHTS_WORKSPACE=1 OPINSIGHTS_WORKSPACE_ID=Your-WorkspaceID OPINSIGHTS_WORKSPACE_KEY=Your-PrimaryKEY AcceptEndUserLicenseAgreement=1 /q

  • Deploy the MOMAgent Application to all Windows 10 SCCM Clients

Configure Log Analytic to collect Windows 10 upgrade logs

Configure Log Analytics to collect the system event log and application event log together with the Windows update client event logs.

To collect Event Viewer Log for Log Analytics Workspace

  • Log Analytics WorkspaceAdvanced settings

tantran_17-1599647128726.png

  • Choose Data,
  • Choose Windows Event Logs
  • Type “Application” and click the + button
  • Type “System” and click the + button
  • Type “Microsoft-Windows-DeviceSetupManager/Admin” and click the + button
  • Type “Microsoft-WindowsUpdateClient/Operation” and click the + button
  • Save

LogAnaAdvSet2.jpg

To collect Windows Upgrade logs for Log Analytics

There are 4 Windows 10 upgrade phases,

  • Downlevel phase: prepare upgrade installer source and destination in Windows 10 OS current version.
  • SafeOS phase, WinPE running phase, copying file for setup, prepare disk and file system table if needed, …
  • Firstboot phase, Windows system driver installation and reboot.
  • Secondboot phase, New version of Windows 10 OS is running, continue installing software applications and drivers.

Depend on the Windows 10 upgrade phases the same upgrade log name could be in different Windows directory locations as shown here, The $WINDOWS~BT path is not working in Log Analytics service:

Log file name Location Suggestions
setupact.log $Windows.~BTSourcesPanther All down-level failures and rollback investigations
setupact.log $Windows.~BTSourcesPantherUnattendGC OOBE phase rollbacks, 0x4001C, 0x4001D, 0x4001E, 0x4001F
setupact.log $Windows.~BTSourcesRollback Generic rollbacks, 0xC1900101
setupact.log Windows Setup launch failures
setupact.log WindowsPanther Post-upgrade issues
setuperr.log $Windows.~BTSourcesPanther Complete error listing
setuperr.log $Windows.~BTSourcesPantherUnattendGC Complete error listing
setuperr.log $Windows.~BTSourcesRollback Complete error listing
setuperr.log Windows Complete error listing
setuperr.log WindowsPanther Complete error listing
miglog.xml WindowsPanther Post-upgrade issues
BlueBox.log WindowsLogsMosetup WSUS and WU down-level failures
setupapi.dev.log $Windows.~BTSourcesRollback Device install issues
setupapi.dev.log C:Windowsinf Complete Device install issues
Setupapi.app.log C:Windowsinf PNP information about operations that install devices and drivers
  • Go to DataCustom logtantran_23-1599647128650.png
  •  Click Add, and “Choose File” button to browse to the log directories specified in the above table.

tantran_25-1599647128708.png

  • Continue add all the logs and path as shown:

tantran_28-1599647128656.png

  • Enter Name of log collection CL, no space allowed in Name.

tantran_29-1599647128658.png

  • Done

tantran_30-1599647128659.png

You may get a permission error, and you would need to “enable inheritance” permission as shown:

LogAnaAdvSet.jpg

Using Kusto Queries to Monitor and Troubleshoot the Upgrade Process

All the search for upgrade status, update compliance status, Windows update delivery optimizaton information could be done by one tool, the Analytics Workspace Log Query as shown:

LogAna.jpg

To Review Update Log and search for errors

Run Log Analytics Query to search for update error in windows logs of devices:

  • In Azure Portal, Log Analytics Workspace
  • Logs, click on + to create new query

UpdateLogs.jpg

  • Choose the Custom Logs, double click to insert the related log to Query Windows
  • Run Query

Query Custom Logs for All Upgrade errors

CompleteWindowsSetupLog_CL

| where TimeGenerated <= ago(24m)

| where RawData contains “error”

tantran_42-1599647128728.png

Query Custom Logs for Upgrade Device Driver Error

PNPDeviceError_CL

| where RawData contains “failure”

tantran_43-1599647128732.png

Query Custom Logs for Upgrade OOBE and other setup error

WindowsUpdatePhaseGC_CL

| where RawData contains “error”

| where RawData contains “WimBoot” or RawData contains “OOBE” or RawData contains “

| project TimeGenerated, Computer, RawData, Type

tantran_44-1599647128735.png

To Search Event Log for Update Errors

Query System Event for Update information:

Event

|where TimeGenerated > ago(1d)

| where EventLog contains “system”

| where RenderedDescription contains “Update”

| project TimeGenerated, EventLog, Computer, EventID

tantran_45-1599647128717.png

To Search Update Compliance Log for Ugrade Errors

Update Compliance Log provide the pre-built Desktop Analytics queries for all status of update included the following:

  • Deployment failures,
  • Reboot pending,
  • Feature or quality update deferral/pause,
  • Update automatic hold by Windows 10 Safeguard (to prevent hardware or software incompatibilities)

tantran_48-1599647128719.png

In Log Analytics Workspace, there is very useful Update Compliance Logs' tables related to WaaS and Windows Update Delivery Optimization as shown:

LA001.jpg

The Update Compliance Logs Query might become our best option in troubleshooting Endpoint Manager Update Feature Deployment.

  • Query WaaS for Windows 10 Upgrade with Not up-to-date Status:

WaaSUpdateStatus

| where OSFeatureUpdateStatus contains “Not Up-to-date”

| project Computer, LastScan, OSName, OSVersion, FeatureDeferralDays, FeaturePauseState, NeedAttentionStatus

tantran_54-1599647128721.png

  • Query WaaS for Upgrade Deployment with error:

WaaSDeploymentStatus

| where DeploymentErrorCode != “0”

  • Query WaaS for Feature Update Deployment not successful, listed by Computer name, Last Scan time, Deployment Status, DetailedStatus…

The DetailedStatus column may show recent feature of Windows 10 2004 with “Safeguard Hold” in the column, Safeguard hold was used to prevent imcompatible device hardware from being upgraded.

WaaSDeploymentStatus

| where DetailedStatus != “UpdateSuccessful”

| where UpdateCategory == “Feature”

| project Computer, LastScan, DeploymentStatus, DeploymentErrorCode, DetailedStatus

  • List Feature Update and Quality Update Status of a specific Computer:

WaaSUpdateStatus

| where Computer == “YourComputerName” and TimeGenerated > ago(30d)

| summarize arg_max(TimeGenerated,OSFeatureUpdateStatus, OSQualityUpdateStatus, NeedAttentionStatus, OSVersion) by Computer

  • Query WaaS for Upgrade Deployment with Failed Status and not contain a specific error code:

WaaSDeploymentStatus

| where TimeGenerated > ago(7d)

| where UpdateCategory == ‘Feature'

| where UpdateClassification == ‘Upgrade'

| where DeploymentStatus == ‘Failed'

| where DeploymentErrorCode notcontains “8007001F”

| where DeploymentError == “N/A”

| where PauseState != “”

Export result to csv file for later investigation of update failure's root cause.

csv.jpg

After we get update error code from query result, then we will need to translate the codes into meaningful root cause description by using the reference table from the following link: https://docs.microsoft.com/en-us/windows/deployment/update/windows-update-error-reference

Continue troubleshooting and testing the deployment of feature updates until we get no error in Log Analytics Query.

You could generate an Intune's update report-workbook as suggested by Jeff Gilbert blog

References

Windows update logs file:

Installing Log Analytics Agent for Windows Computers:

Update Compliance Setup and Log Analytics Queries:

Kusto Query tips and examples:

I hope the information is useful for Windows Feature Update Troubleshooting.

On my next blogpost, we will re-visit and discuss Update Compliance again.

Cheers!


Disclaimer

The sample are not supported under any Microsoft standard support program or service. The sample are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.

 

This article was originally published by Microsoft's Core Infrastructure and Security Blog. You can find the original article here.