The Mysterious Case of the Self-Moving FSMO Roles

>Hello all>! >This is Chris Cartwright from Directory Services>. >I had a coworker>,Eric Jansen>,>reach out to me from the>field and>ask about an incident on site he was looking into> a scenario>where “>the PD>CE (Primary Emulator)>and>DNM (>Domain Naming Master>)>mysteriously moved…” to>a>DC>in another site>. >He said what was weird was>who>the logs said performed it>. >He also said that the other site used their own procedures to build their>DCs>, which>apparently included>usingWindows Servers Essentials>for the base>OS>. >Now,>I have>never heard of anyone doing that>in an enterprise environment>, but it got us curious… 

  

Administrators are running the beautiful, pristine Contoso domain forest, (with 2 DCs of course…because two is one, and one is none!).  They were built with 2012 R2 Datacenter.  They know that the FSMOs are still on the first DCs built, and can see that by running “netdom query fsmo“:    

EJansen_0-1608667230286.png 

  

One day, Contoso-HQ has a new subsidiary, Tailspin Toys that needs services for their store.  Per Contoso's Organizational Policy, this company will exist on the same domain, and they have authorized admins at that site to promote the new DCs for that site.  In preparation for this, the Contoso admins create the site and subnets for the new company's location.   

EJansen_1-1608667230289.png

30 days go by and the admins at Contoso notice something odd: 

EJansen_2-1608667230292.png

The problem is…nobody authorized that change.  They call the admins at Tailspin Toys and asked if they knew anything about this, and they didn't…. Not good.   

So, the first thing that they did was figure out when the change occurred.  To do that they used Repadmin.exe with the /ShowObjMeta switch, to see when the FSMORoleOwner attribute changed for the roles that they were interested in.   The timestamp for the last modification to that attribute can be found by looking at the object metadata for that attribute at the following DN locations (for each respective role): 

  

  • RID Master – “cn=Rid Manager$,cn=system,dc=contoso,dc=com”
  • Infrastructure Master – “cn=infrastructure, dc=contoso,dc=com”
  • PDC Emulator – “dc=contoso,dc=com“
  • Domain Naming Master – “cn=partitions,dc=configuration, dc=contoso,dc=com“
  • Schema Master – “cn=schema,dc=configuration, dc=contoso,dc=com“

In this case, they were interested in the Domain Naming Master, and the PDCE Emulator role movements operations. 

EJansen_3-1608667230214.png

Once the Contoso Admins figured out when the role was moved and where the change originated from, they now know where to start their search in the event logs.  With that said, the Contoso admins transfer the roles back, and then start digging through the logs.   

They find these two events in the Directory Services logs on ContosoDC1: 

(PDC) 

EJansen_4-1608667230315.png 

(Domain Naming Master) 

EJansen_5-1608667230317.png

 So, the change did come from ContosoDC3, but the admins know that they can look at the logs on DC3 to see what user account initiated this, because it's listed with “User.” 

So, they take a look at ContosoDC3…but what they see isn't exactly what they expected: 

EJansen_6-1608667230306.png

  

EJansen_7-1608667230308.png

   

The user is SYSTEM.  Has ContosoDC3 become self-aware?  Should we expect robots from the future?  The administrators continue digging, now focusing on the security logs during the same timeframe.  At 8:54:35 PM, the same time that they saw the roles move in the Directory Services log, they now find the following in the security logs on ContosoDC3: 

Note the Login ID… 

EJansen_8-1608667230310.png

  

EJansen_9-1608667230303.png

This log entry continued… 

EJansen_10-1608667230295.png

  

This event looks interesting.  “An operation was performed on an object.”  The operation in this case was a “Change PDC Operation” (https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/99dd371d-9ede-417f-beb1-a065c10ef68f

EJansen_11-1608667230311.png

This just goes to show how important it is to have proper auditing in place...you DO have proper auditing setup, enabled, and verified in your environment, don't you?  If not, I suggest you take a look at Michael Hildebrand's blog that discusses this in more detail, here.     

So, what happened?  Well, answering that would have taken a little more logging.  Procmon, for example, would have shown that the source port used above came from the silsvc.exe process on the same server.   

EJansen_12-1608667230313.png

  

So, what is this silsvc.exe you ask?  That's the Server Infrastructure Licensing Service, and fortunately, it has its own log: 

EJansen_13-1608667230319.png

  

And the final nail in the coffin is the very next event – “The Correction” 

EJansen_14-1608667230299.png

To sum everything up, do not put Windows Server Essentials into an existing large enterprise – it was never meant to co-exist, and all of the folks (at least that I talked to in both CSS and PFE) had never seen this scenario before.  In the real-world scenario where this happened, those remote ‘spoke site' DC's were only meant to be stood up temporarily, and they were.  They were stood up just long enough for the compliance threshold to be met, which moved the roles unknowingly to the ‘spoke site', and the next day those DC's were taken offline permanently.  It just happened that those DC's were removed on a Friday afternoon and the repercussions weren't felt until the following week. 

We tested this with Server 2012 R2 Essentials and Essentials while manually moving the time forward and saw the same results.   Essentials has the same warning, however we were unable to reproduce the issue just by simply moving the time 30 days into the future: 

EJansen_15-1608667230252.png

So, it's being tested the old-fashioned way, and we'll update on that later in 2021! 

 

This article was originally published by Microsoft's Core Infrastructure and Security Blog. You can find the original article here.