THIS JUST IN!!!!  High LSASS Usage After Windows Update 3B March 2024

Jim and the Directory Services Team here again to alert you to an emerging issue which is an unintended consequence of a recent update released in March 2024.  

What is LSASS and why is it important? 

The Local Security Authority Subsystem Service (LSASS) is a process that handles user , security policies, and auditing on Windows systems. It is essential for the proper functioning of your computer, as it verifies your identity and facilitates your access to your files and applications.  For domain controllers, it has the additional responsibility of hosting the related services that provide authentication, , database query processing, and other domain functions. 

Given the importance of the LSASS process, most Enterprise environments monitor its operation and alert when LSASS is consuming a large amount of CPU or memory resources affecting the system's performance. This can happen due to assorted reasons, but in this blog post, we will focus on one specific cause that has been recently reported and is currently being addressed by the Microsoft Product Group. 

What is the 3B Windows update and how does it affect LSASS? 

As of March 18, 2024, customers are experiencing excessive memory consumption by LSASS on Windows Server 2012-2022 DCs that have installed the following Windows Update(s): 

KB 5035857: March 12, 2024, KB5035857 (OS Build 20348.2340) Windows Server 2022 

KB 5035849: March 12, 2024, KB5035849 (OS Build 17763.5576) Windows Server 2019 
KB 5035855: March 14, 2024, KB5035855 (OS Build 14393.5786) Windows

KB 5035885: March 12, 2024, KB5035885 Monthly Rollup for Windows Server 2012 R2: March 12, 2024 

Affected platforms: Windows Server 2022, Windows Server 2019, Windows , Windows Server 2012 R2 
 

Following installation of the March 2024 security updates released March 12, 2024, the Local Security Authority Subsystem Service (LSASS) may experience a memory leak on domain controllers (DCs). This is observed when on-premises and cloud-based Domain Controllers service authentication service requests (AS-REQ). 

SYMPTOMS 

Log Name: System 

Source: Microsoft-Windows-Resource-Exhaustion-Detector 

Event ID: 2004 

Task Category: Resource Exhaustion Diagnosis Events 

Level: Warning 

Keywords: Events related to exhaustion of system commit limit (virtual memory). 

User: SYSTEM 

Computer: Description: 
Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: lsass.exe (PID) consumed bytes, . (PID ) consumed bytes, and . (PID) consumed bytes. 

 
Alternatively, if you have other resource monitoring software, you may want to leverage it for restarts to keep in line with organizational requirements and procedures. 

LSASS Memory leaks at the rate of 2GB per hour have been observed. Memory exhaustion may cause application or service crashes, including the crashing of LSASS which in turn will trigger a reboot of the underlying OS. In addition, customers who have very busy domain controllers will experience not only the memory leak, but these sorts of heap leaks in LSASS typically also cause a lot of heap fragmentation.  This heap fragmentation can cause a surprisingly severe CPU performance penalty in addition to just memory growth.  The high CPU usage may be the first performance indicator seen and could be indicative of the underlying memory leak problem. 

LSASS Private Bytes increases linearly with system uptime: 

JIMT05_2-1711403998230.png

For more information, see Use Performance Monitor to Find a User-Mode Memory Leak – Windows drivers | Microsoft Learn. 

Task Manager 

Task Manager shows LSASS consuming significant percentage of memory: 
 

JIMT05_3-1711403998231.png

Lsass.exe Process Exceptions  

LSASS crashes and reboots the entire server after LSASS consumes sufficient memory.  LSASS crashes and device reboots will occur more often on physical and with LESS memory. 

Associated event log entries: 

Log Name: Application  
Source: Application Error  
Event ID 1000:   
Faulting application name: lsass.exe, version: 6.3.9600.17415, time stamp: 0x545042fe  
Faulting module name: .DLL, version: 6.3.9600.17423, time stamp: 0x545ff681  
Exception code: 0xc0000005  
Fault offset: 0x00000000000910b7  
Faulting process id: 0x448  
Faulting application start time: 0x01d029e23a389f2e  
Faulting application path: C:Windowssystem32lsass.exe  
Faulting module path: C:Windowssystem32kerberos.DLL  
 

Log Name:      System 

Source:        User32 

Event ID:      1074 

User:          SYSTEM 

Description: 

The process wininit.exe has initiated the restart of computer on behalf of user for the following reason: No title for this reason could be found 

Reason Code: 0x50006 

Shutdown Type: restart 

Comment: The system process ‘C:WINDOWSsystem32lsass.exe' terminated unexpectedly with status code -1073741819.  The system will now shut down and restart. 

Log Name: Application   
Source: Microsoft-Windows-Wininit   
Date: DateTime  
Event ID: 1015   
Task Category: None   
Level: Error   
Keywords: Classic   
User: N/A   
Computer: ComputerName   
Description:   
A critical system process, C:WINDOWSsystem32lsass.exe, failed with status code c0000005.  The machine must now be restarted. 

How to fix high LSASS usage after Windows update? 

Updated 4/15/2024  

We have seen some questions in the comments around whether the April 2024 Cumulative/Security updates do or do not include the Out of Band release for the March 2024 update.  The short answer is yes, the OOB update is superseded by the April cumulative Update, and at this time you can skip the March updates if you have not applied them as of yet and install just the April 2024 update if you would like.

The commenters are correct that the release notes do not specifically list this update.  So how do you know that it has been superseded, and more importantly how can windows administrators find out this information without relying on forums or opening a support case? Well, fortunately    you can do this by looking at the Microsoft Update catalog at the address  https://aka.ms/updatecatalog

1.  Once you get to the page you can type in the KB number in the search bar that you are interested in. 

RobGreene_0-1713195472569.png

2.  Once you get the result back showing you the update that you searched on, click on the “Title” of the update. 

RobGreene_1-1713195472572.png

3.  It will spawn a context browser showing you information about the specific update.  We are interested in the Package Details tab on this page.   

RobGreene_2-1713195472574.png

 4.  The Red box shows what updates are replaced by this update, while the Green box shows what updates replace this update. 

 

So, by looking at the information about the Windows Server 2022 March 2024 OOB (out of band) update Catalog site we can determined that this update has been superseded by April 2024 Cumulative update. 

Background details and previous guidance 

How long until your domain controller begins to experience failures after the March update is installed varies based on how much RAM is available to it, and how much authentication traffic is being sent to it.  If it is critical to have your DC's reboot before running out of memory, an Event Trigger for Event ID 2004 could be configured to reboot the server when that event is logged if that would help.  
However, if your DC's have a large amount of memory, you may just want to perform proactive periodic reboots of your domain controllers before they hit their maximum memory range. 

Fortunately, there are workable solutions that you can use to address the high LSASS usage after the 3b Windows update has been installed.  See the following FIRST for installation details and methodologies – 
https://learn.microsoft.com/en-us/windows/release-health/status-windows-server-2022#march-2024 
 
The root cause has been identified and the current resolution is an Out-of-band update (OOB) available as of NOW!!!
The OOB update is available via the Windows Catalog location links below.  The OOB update will NOT be available through the normal Windows update channels. 

Server 2022 3OOB: March 22, 2024—KB5037422 (OS Build 20348.2342) Out-of-band – Microsoft Support 
Windows Server 2019 3OOB: March 25, 2024—KB5037425 (OS Build 17763.5579) Out-of-band – Microsoft Support 
3OOB: March 22, 2024—KB5037423 (OS Build 14393.6799) Out-of-band – Microsoft Support 
SupportServer 2012 R2 3OOB: KB5037426: Update to address a known issue that affects LSASS in Windows Server 2012 R2 – Microsoft …   
  

Download the aforementioned OOB update from the links provided above for your operating system and install. 
You do not have to uninstall the 3b update prior to installing the OOB update.  If you have not installed the 3b update you can just install the OOB update instead. 

Uninstalling the 3b Windows update is not recommended.  Although this may seem like the most straightforward and effective way to resolve the issue, your servers will remain vulnerable to multiple bug fixes and other CVEs that ship in the average Monthly Updates. 

Jim “looking forward to the next update :sad:” Tierney and the DS Gang!  

 

This article was originally published by Microsoft's Directory Services Team. You can find the original article here.