Troubleshooting failed password changes after installing MS16-101

First published on TechNet on Oct 13, 2016


Hi!

Linda Taylor here, Senior Escalation Engineer in the Directory Services space.

I have spent the last month working with customers worldwide who experienced password change failures after installing the updates under Ms16-101 security bulletin KB's (listed below), as well as working with the product group in getting those addressed and documented in the public KB articles under the

known issues

section. It has been busy!

In this post I will aim to provide you with a quick “cheat sheet” of known issues and needed actions as well as ideas and troubleshooting techniques to get there.

Let's start by understanding the changes.

The following 6 articles describe the changes in MS16-101 as well as a list of

Known issues.

If you have not yet applied MS16-101 I would strongly recommend reading these and understanding how they may affect you.

3176492

Cumulative update for Windows 10: August 9, 2016


3176493

Cumulative update for Windows 10 Version 1511: August 9, 2016


3176495

Cumulative update for Windows 10 Version 1607: August 9, 2016


3178465

MS16-101: Security update for Windows authentication methods: August 9, 2016


3167679

MS16-101: Description of the security update for Windows authentication methods: August 9, 2016


3177108

MS16-101: Description of the security update for Windows authentication methods: August 9, 2016

The good news





is

that this month's updates address some of the known issues with MS16-101.

The bad news


is

that not all the issues are caused by some code defect in MS16-101 and in some cases the right solution is to make your environment more secure by ensuring that the password change can happen over Kerberos and does not need to fall back to NTLM. That may include opening TCP ports used by Kerberos, fixing other Kerberos problems like missing SPN's or changing your application code to pass in a valid domain name.

Let's start with the basics…

Symptoms:

After applying MS16-101 fixes listed above, password changes may fail with the error code

“The system detected a possible attempt to compromise security. Please make sure that you can contact the server that authenticated you.”

Or

“The system cannot contact a domain controller to service the authentication request. Please try again later.”

This text maps to the error codes below:

Question: What does MS16-101 do and why would password changes fail after installing it?

Answer:


As documented in the listed KB articles, the security updates that are provided in MS16-101 disable the ability of the

Microsoft Negotiate SSP

to fall back to NTLM for password change operations in the case where Kerberos fails with the STATUS_NO_LOGON_SERVERS (0xc000005e) error code.

In this situation, the password change will now fail (post MS16-101) with the above mentioned error codes (ERROR_DOWNGRADE_DETECTED / STATUS_DOWNGRADE_DETECTED).



Important:


Password RESET is not affected by MS16-101 at all in any scenario. Only password change using the Negotiate package is affected.

So, now you understand the change, let's look at the known issues and learn how to best identify and resolve those.

Summary and Cheat Sheet

To make it easier to follow I have matched the ordering of known issues in this post with the public KB articles above.

First, when troubleshooting a failed password change post MS16-101 you will need to understand

HOW

and

WHERE

the password change is happening and if it is for a domain account or a local account. Here is a cheat sheet.

Summary of SCENARIO's and a quick reference table of actions needed.

Table of Fixes for known issues above release 2016.10.11, taken from

MS16-101 Security Bulletin

:

Troubleshooting

As I mentioned, this post is intended to support the documentation of the

known issues

in the Ms16-101 KB articles and provide help and guidance for troubleshooting. It should help you identify which known issue you are experiencing as well as provide resolution suggestions for each case.

I have also included a troubleshooting walkthrough of some of the more complex example cases. We will start with the problem definition, and then look at the available logs and tools to identify a suitable resolution. The idea is to teach “how to fish” because there can be many different scenario's and hopefully you can apply these techniques and use the log files documented here to help resolve the issues when needed.

Once you know the scenario that you are using for the password change the next step is usually to collect some data on the server or client where the password change is occuring. For example if you have a web server running a password change application and doing password changes on behalf of users, you will need to collect the logs there. If in doubt collect the logs from all involved machines and then look for the right one doing the password change using the snippets in the examples. Here are the helpful logs.

DATA COLLECTION

The same logs will help in all the scenario's.

LOGS

1

. SPENGO debug log/ LSASS.log

To enable this log run the following commands from an elevated admin CMD prompt to set the below registry keys:

• This will log Negotiate debug output to the %windir%system32
  
  lsass.log.
  
• There is no need for reboot. The log is effective immediately.
• Lsass.log is a text file that is easy to read with a text editor such as Wordpad.

2. Netlogon.log:

This log has been around for many years and is useful for troubleshooting DC LOCATOR traffic. It can be used together with a network trace to understand why the STATUS_NO_LOGON_SERVERS is being returned for the Kerberos password change attempt.

· To enable Netlogon debug logging run the following command from an elevated CMD prompt:

nltest /dbflag:0x26FFFFFF

· The resulting log is found in %windir%debug

netlogon.log



netlogon.bak

· There is no need for reboot. The log is effective immediately. See also

109626

Enabling debug logging for the Net Logon service

· The

Netlogon.log

(and Netlogon.bak) is a text file.

Open the log with any text editor (I like good old Notepad.exe)

3. Collect a


Network trace


during the password change issue using the tool of your choice.

Scenario's, Explanations and Walkthrough's:

When reading this you should keep in mind that you may be seeing more than one scenario. The best thing to do is to start with one, fix that and see if there are any other problems left.

1. Domain password change fails via CTRL+ALT+DEL

This is most likely a Kerberos DC locator failure of some kind where the password changes were relying on NTLM before installing MS16-101 and are now failing. This is the simplest and easiest case to resolve using basic Kerberos troubleshooting methods.

Solution:



Fix Kerberos.

Some tips from cases which we saw:

1. Use the Network trace to identify if the necessary communication ports are open. This was quite a common issue. So start by checking this.

In order for Kerberos password changes to work communication on TCP port 464 needs to be open between the client doing the

password change and the domain controller.

Note on RODC: Read-only domain controllers (RODCs) can service password changes if the user is allowed by the RODCs password replication policy. Users who are not allowed by the RODC password policy require network connectivity to a read/write domain controller (RWDC) in the user account domain to be able to change the password.

To check whether TCP port 464 is open, follow these steps (also documented in KB



3167679



):

a. Create an equivalent display filter for your network monitor parser. For example:

ipv4.address==

b. In the results, look for the “TCP:[SynReTransmit” frame.

If you find these, then investigate firewall and open ports. It is often useful to take a simultaneous trace from the client and the domain controller and check if the packets are arriving at the other end.

2. Make sure that the target Kerberos names are valid.

• IP addresses are not valid Kerberos names
• Kerberos supports short names and fully qualified domain names. Like CONTOSO or Contoso.com

3. Make sure that service principal names (SPNs) are registered correctly.

For more information on troubleshooting Kerberos see

https://blogs.technet.microsoft.com/askds/2008/05/14/troubleshooting-kerberos-authentication-pro…

https://technet.microsoft.com/en-us/library/cc728430(v=ws.10).aspx


2. Domain password change fails via application code with an INCORRECT/UNEXPECTED Error code when a password which does not meet password complexity is entered.

For example, before installing MS16-101, such password change may have returned a status like STATUS_PASSWORD_RESTRICTION. After installing Ms16-101 it returns STATUS_DOWNGRADE_DETECTED causing your application to behave in an expected way or even crash.

Note: In this scenario, password change succeeds when correct new password is entered that complies with the password policy.

Cause:

This issue is caused by a code defect in ADSI whereby the status returned from Kerberos was not returned to the user by ADSI correctly.

Here is a more detailed explanation of this one for the geek in you:

Before MS16-101 behavior:

1. An application calls ChangePassword method from using the ADSI LDAP provider.

Setting and changing passwords with the ADSI LDAP Provider is documented

here.


Under the hood this calls Negotiate/Kerberos to change the password using a valid realm name.

Kerberos returns STATUS_PASSWORD_RESTRICTION or Other failure code.

2. A 2nd changepassword call is made via NetUserChangePassword API with an intentional realmname as the which uses

Negotiate and will retry Kerberos. Kerberos fails with STATUS_NO_LOGON_SERVERS because a DC name is not a valid realm name.

3.Negotiate then retries over NTLM which succeeds or returns the same previous failure status.

The password change fails if a bad password was entered and the NTLM error code is returned back to the application. If a valid password was entered, everything works because the 1st change password call passes in a good name and if Kerberos works, the password change operation succeeds and you never enter into step 3.

Post MS16-101 behavior /why it fails with MS16-101 installed:

1. An application calls ChangePassword method from using the ADSI LDAP provider. This calls Negotiate for the password change with

a valid realm name.

Kerberos returns STATUS_PASSWORD_RESTRICTION or Other failure code.

2. A 2nd ChangePassword call is made via NetUserChangePassword with a as realm name which fails over Kerberos with

STATUS_NO_LOGON_SERVERS which triggers NTLM fallback.

3. Because NTLM fallback is blocked on MS16-101, Error STATUS_DOWNGRADE_DETECTED is returned to the calling app.

Solution:


Easy. Install the October update which will fix this issue. The fix lies in


adsmsext.dll


included in the October updates.

Again, here are the updates you need to install, Taken from

MS16-101 Security Bulletin

:

3.Local user account password change fails via CTRL+ALT+DEL or application code.

Installing October updates above should also resolve this.

MS16-101 had a defect where Negotiate did not correctly determine that the password change was local and would try to find a DC using the local machine as the domain name.

This failed and NTLM fallback was no longer allowed post MS16-101. Therefore, the password changes failed with STATUS_DOWNGRADE_DETECTED.

Example:

One such scenario which I saw where password changes of local user accounts via ctrl+alt+delete failed with the message “The system detected a possible attempt to compromise security. Please ensure that you can contact the server that authenticated you.” Was when you have the following group policy set and you try to change a password of a local account:

Cause:


In this case, post MS16-101 Negotiate incorrectly determined that the account is not local and tried to discover a DC using as the domain and failed. This caused the password change to fail with the STATUS_DOWNGRADE_DETECTED error.

Solution:


Install October fixes listed in the table at the top of this post.



4.Passwords for disabled and locked out user accounts cannot be changed using Negotiate method.

MS16-101 purposely disabled changing the password of locked-out or disabled user account passwords via Negotiate by design.

Important: Password Reset is not affected by MS16-101 at all in any scenario. Only password change. Therefore, any application which is doing a password Reset will be unaffected by Ms16-101.

Another important thing to note is that MS16-101 only affects applications using Negotiate. Therefore, it is possible to change locked-out and disabled account password using other method's such as LDAPs.

For example, the PowerShell cmdlet


Set-ADAccountPassword


will continue to work for locked out and disabled account password changes as it does not use Negotiate.



5. Troubleshooting Domain password change failure via application code when a good password is entered.

This is one of the most difficult scenarios to identify and troubleshoot. And therefore I have provided a more detailed example here including sample code, the cause and solution.

In summary,

the solution for these cases is almost always to correct the application code which maybe passing in an invalid domain name such that Kerberos fails with STATUS_NO_LOGON_SERVERS.

Scenario:

An application is using

system.directoryservices.accountmanagement

namespace to change a users password.


https://msdn.microsoft.com/en-us/library/system.directoryservices.accountmanagement(v=vs.110).as…

After installing Ms16-101 password changes fail with STATUS_DOWNGRADE_DETECTED. Example .NET failing code snippet using PowerShell which worked before MS16-101:

Add-Type -AssemblyName System.DirectoryServices.AccountManagement

$ct = [System.DirectoryServices.AccountManagement.ContextType]::Domain

$ctoptions = [System.DirectoryServices.AccountManagement.ContextOptions]::SimpleBind -bor [System.DirectoryServices.AccountManagement.ContextOptions]::ServerBind

$pc = New-Object System.DirectoryServices.AccountManagement.PrincipalContext($ct, “contoso.com”,”OU=Accounts,DC=Contoso,DC=Com”, ,$ctoptions)

$idType = [System.DirectoryServices.AccountManagement.IdentityType]::SamAccountName

$up = [System.DirectoryServices.AccountManagement.UserPrincipal]::FindByIdentity($pc,$idType, “TestUser”)

$up.ChangePassword(“oldPassword!123”, “newPassword!123”)

Data Analysis

There are 2 possibilities here:


(a)

The application code is passing an incorrect domain name parameter causing Kerberos password change to fail to locate a DC.


(b)

Application code is good and Kerberos password change fails for other reason like blocked port or DNS issue or missing SPN.

Let's start with

(a)

The application code is passing an incorrect domain name/parameter causing Kerberos password change to fail to locate a DC.

(a) Data Analysis Walkthrough Example based on a real case:


1. Start with Lsass.log (SPNEGO trace)

If you are troubleshooting a password change failure after MS16-101 look for the following text in

Lsass.log

to indicate that Kerberos failed and NTLM fallback was forbidden by Ms16-101:

Failing Example:

[ 9/13 10:23:36] 492.2448> SPM-WAPI: [11b0.1014] Dispatching API (Message 0)

[ 9/13 10:23:36] 492.2448> SPM-Trace: [11b0] LpcDispatch: dispatching ChangeAccountPassword (1a)

[ 9/13 10:23:36] 492.2448> SPM-Trace: [11b0] LpcChangeAccountPassword()

[ 9/13 10:23:36] 492.2448> SPM-Helpers: [11b0] LsapCopyFromClient(0000005EAB78C9D8, 000000DA664CE5E0, 16) = 0

[ 9/13 10:23:36] 492.2448> SPM-Neg: NegChangeAccountPassword:

[ 9/13 10:23:36] 492.2448> SPM-Neg: NegChangeAccountPassword, attempting: NegoExtender

[ 9/13 10:23:36] 492.2448> SPM-Neg: NegChangeAccountPassword, attempting: Kerberos

[ 9/13 10:23:36] 492.2448> SPM-Warning: Failed to change password for account Test: 0xc000005e

[ 9/13 10:23:36] 492.2448> SPM-Neg: NegChangeAccountPassword, attempting: NTLM

[ 9/13 10:23:36] 492.2448> SPM-Neg: NegChangeAccountPassword, NTLM failed: not allowed to change domain passwords

[ 9/13 10:23:36] 492.2448> SPM-Neg: NegChangeAccountPassword, returning: 0xc0000388

• 0xc000005E is STATUS_NO_LOGON_SERVERS
  
  0xc0000388 is STATUS_DOWNGRADE_DETECTED

If you see this, it means Kerberos failed to locate a Domain Controller in the domain and fallback to NTLM is not allowed by Ms16-101. Next you should look at the

Netlogon.log

and the

Network trace

to understand why.

2. Network trace

Look at the network trace and filter the traffic based on the client IP, DNS and any authentication related traffic.

You may see the client is requesting a Kerberos ticket using an invalid SPN like:

So here the client tried to get a ticket for this ldapContoso.com SPN and failed with KDC_ERR_S_PRINCIPAL_UNKNOWN because this SPN is not registered anywhere.

• This is expected. A valid LDAP SPN is example like ldapDC1.contoso.com

Next let's check the Netlogon.log

3. Netlogon.log:

Open the log with any text editor (I like good old Notepad.exe) and check the following:

• 
  
  Is a valid domain name being passed to DC locator?
  
  

Invalid names such as

servername.contoso.com

or IP address

x.y.x.w

will cause dclocator to fail and thus Kerberos password change to return STATUS_NO_LOGON_SERVERS. Once that happens NTLM fall back is not allowed and you get a failed password change.

If you find this issue examine the application code and make necessary changes to

ensure correct domain name format is being passed to the ChangePassword API

that is being used.

Example of failure in Netlogon.log:

[MISC] [PID] DsGetDcName function called: client PID=1234, Dom:contoso.com Acct:(null) Flags: IP KDC

[MISC] [PID] DsGetDcName function returns 1212 (client PID=1234): Dom:contoso.com Acct:(null) Flags: IP KDC

contoso.com is not a valid domain name. (contoso.com is a valid domain name)

This Error translates to:

So what happened here?

The application code passed an invalid TargetName to kerberos. It used the domain name

as a server name

and so we see the SPN of LDAPcontoso.com.

The client tried to get a ticket for this SPN and failed with KDC_ERR_S_PRINCIPAL_UNKNOWN because this SPN is not registered anywhere. As Noted: this is expected. A valid LDAP SPN is example like ldap

DC1

.contoso.com.

The application code then tried the password change again and passed in

contoso.com

as a domain name for the password change. Anything beginning with as domain name is not valid. IP address is not valid. So DCLOCATOR will fail to locate a DC when given this domain name. We can see this in the Netlogon.log and the Network trace.

Conclusion and Solution

If the domain name is invalid here, examine the code snippet which is doing the password change to understand why the wrong name is passed in.

The fix in these cases will be to change the code to ensure a valid domain name is passed to Kerberos to allow the password change to successfully happen over Kerberos and not NTLM. NTLM is not secure. If Kerberos is possible, it should be the protocol used.

SOLUTION

The solution here was to remove “ContextOptions.

ServerBind

|  ContextOptions.SimpleBind ” and allow the code to use the default (Negotiate). Note, because we were using a domain context but ServerBind this caused the issue. Negotiate with Domain context is the option that works and is successfully able to use kerberos.

Working code:

Add-Type -AssemblyName System.DirectoryServices.AccountManagement

$ct = [System.DirectoryServices.AccountManagement.ContextType]::Domain

$pc = New-Object System.DirectoryServices.AccountManagement.PrincipalContext($ct, “contoso.com”,”OU=Accounts,DC=Contoso,DC=Com”)

$idType = [System.DirectoryServices.AccountManagement.IdentityType]::SamAccountName

$up = [System.DirectoryServices.AccountManagement.UserPrincipal]::FindByIdentity($pc,$idType, “TestUser”)

$up.ChangePassword(“oldPassword!123”, “newPassword!123”)



Why does this code work before MS16-101 and fail after?

ContextOptions

are documented here:

https://msdn.microsoft.com/en-us/library/system.directoryservices.accountmanagement.contextoptio…

Specifically:

“This parameter specifies the options that are used for binding to the

server.

The application can set multiple options that are linked with a bitwise OR operation. “

Passing in a domain name such as

contoso.com

with the ContextOptions ServerBind or SimpleBind causes the client to attempt to use an SPN like

ldapcontoso.com

because it expects the name which is passed in to be a ServerName.

This is not a valid SPN and does not exist, therefore this will fail and as a result Kerberos will fail with STATUS_NO_LOGON_SERVERS.

Before MS16-101, in this scenario, the Negotiate package would fall back to NTLM, attempt the password change using NTLM and succeed.

Post MS16-101 this fall back is not allowed and Kerberos is enforced.

(b) If Application Code is good but Kerberos fails to locate a DC for other reason

If you see a correct domain name and SPN's in the above logs, then the issue is that kerberos fails for some other reason such as blocked TCP ports. In this case revert to Scenario 1 to troubleshoot why Kerberos failed to locate a Domain Controller.

There is a chance that you may also have both (a) and (b). Traces and logs are the best tools to identify.

Scenario6: After you install MS 16-101 update, you may encounter 0xC0000022 NTLM authentication errors.

I will not go into detail of this scenario as it is well described in the KB article

KB3195799

NTLM authentication fails with 0xC0000022 error for Windows Server 2012, Windows 8.1, and Windows Server 2012 R2 after update is applied.

That's all for today! I hope you find this useful. I will update this post if any new information arises.

Linda Taylor | Senior Escalation Engineer | Windows Directory Services


(A well established member of the content police.)

This article was originally published by Microsoft's Directory Services Team. You can find the original article here.