Hey Everyone. Randy, Premier Field Engineer, here to discuss some lessons learned from working with a recent merger between two corporations. I don’t have enough time or space to go into the details of this major endeavor, so I am going to talk about this experience with a “Top Ten Countdown” style BLOG POST. I’m sure there are many other headaches seen in the field and I would love to hear about them in the comments section. Maybe I can start to consolidate all this into a Wiki about Partnerships and Mergers between two dueling Active Directory environments. Let’s get started:
1. This one being the most important! Never establish multiple trust paths: I have had the same conversation with countless engineers when doing phone support, about setting up both a Forest Trust between the two Forest Roots, and also an External Trust between two child domains in each of the forests. This should never be done under any circumstances. The argument is often “But this is a shortcut trust.” A Shortcut Trust is between two different domains in the same forest. I have also seen arguments where certain applications (here is an example) that are performing logon routines are not able to query a forest, and therefore need a direct trust. There is likely a newer version of the application without this requirement. If there is not an update or competitive product without this requirement, then it is time to do some soul searching on what is more important. The crux of the issue is different technologies providing the trust path between the same domains, each having different characteristics and limitations. One workflow may use the enumeration of trusted domains and hit one of these limitations based on the technology invoked.
2. Beware of CNAME (alias) records in DNS: This is true regardless of traversing a trust, or in the local domain. The behavior of how SPNs are formatted in the client’s request has dramatically changed over various versions of Windows. This article talks about this behavior, although it is not that straight forward about why it is a problem. When accessing a resource using Kerberos Authentication, the client has to construct a Service Principal Name based on the Host Name offering that service. Take a look at the example Below:
Here we have a File Server (FileServ1.adatum.com) hosting a Share (SalesReportsMonthly.) In DNS, we configure a Cname record to direct any DNS query for SalesReports.adatum.com, to the physical host of website Fileserv1.adatum.com.
In Windows XP and SMBv1: We would ask the KDC/Domain Controller for CIFS/SalesReports, because that was the hostname supplied in the UNC Path.
In Vista and SMBv2: We would ask the KDC/Domain Controller for CIFS/FileSrv1.adatum.com, because that was the hostname resolved in the DNS CNAME query.
In Windows 8: We once again returned to the original behavior of asking for CIFS/SalesReports, and adding the “NETDOM COMPUTERNAME /ADD” to supply an alternate name that would register the SPN for you.
The recommendation from Microsoft is to avoid CNAME records if at all possible. This will avoid a variety of headaches because you could see unexpected outcomes as you use other network transports like HTTP.
3. Use Fully Qualified Domain Names: When joining a domain, writing logon scripts, or configuring an application setting that requires a computer or domain name, I have just made this a habit ever since about 2003. There are plenty of ways that Windows can overcome flat names, but why not keep it simple wherever you can. By supplying the FQDN, we tell DNS exactly what we want without confusion. Here is a short list of problems you will avoid:
1. Same Host Names exist in multiple domains
2. Time delays having to parse through the domain suffix search order to look for a match
3. Kerberos KDC knowing which realm to forward the ticket request
4. Disjoined Name Spaces or Unorthodox Naming Conventions
The next two items on the list are related to FQDN.
4. Kerberos Forest Search Order: In some situations, flat names may be used and will format the SPN request made by the client as “HTTP/ServerName” (for example) and not include the domain suffix. The domain suffix is important because the user will always go to its local domain’s KDC which uses the domain suffix to identify which Kerberos Realm it should direct the user. There is a GPO setting that can be configured either for the client or the KDC which lists out other realms where it can check for a matching SPN. A configuration that I found useful is to list the Workstation Forest in the KFSO GPO applied to the workstation. This was helpful in a situation where users have been migrated to the new AD Forest, but their workstations have yet to be migrated. Let’s watch this play out in a cartoon entitled “The Adventures of Joey Adatum!”
The beauty of this solution is that there would be no performance impact (as described in the TechNet documentation) because the workstation is just asking its local Forest and would only be relevant if the logged on user was from a different Forest.
5. DFS Namespaces: To continue on the topic of domain suffixes, update DFSN to use DNS FQDN’s. This will ensure that any cross forest usage of DFS Namespaces will resolve correctly. Another important aspect is the Special Names Table generated for the forest. The Special Names Table is a list stored on each Windows client that it gets from the DC. It is a listing of domain names that the DFS client needs to recognize when resolving a UNC path as either a Domain-Based DFSN or a regular Server Name. A Two-Way Forest trust will not give you any problems with populating this table for the partner forest. If you start playing with One-Way, or External Trusts, you might not get the results you are looking for, especially when child domains are involved. You can see the Special Names Table by using “dfsutil /spcinfo” on a client that has the DFS RSAT tools installed. See the example below when running dfsutil /spcinfo on a workstation in Widgets.Adatum.com:
The good news is Windows 8 & 10 clients seem to behave, and I only found this to be an issue with Windows 7 / 2008 systems. This can be a major problem with new SMB Hardening recommendations for *Sysvol and *Netlogon. If you have a One-Way trust and Windows 7 clients with SMB hardening, then they won’t be getting group policy across the forest because Kerberos will fail.
6. Will users log on interactively or via RDP to systems across the Forest Trust? I am including this as a separate line item because it will likely be the biggest headache that you encounter. The two items above on our list are examples of problems you might face. It would be helpful to really think through these situations and see where Network Access would be sufficient.
7. Don’t forget about udp port 389: Firewall configuration is always important when troubleshooting cross forest failures. Make sure all the required ports are open for Active Directory, udp 389 is often forgotten, but very important for DC Discovery operations.
8. Site Affiliation: For optimal performance, we want to make sure that we are choosing resources located near our physical location. I won’t go into detail, because a great BLOG POST was written eight years ago that still reigns true today.
9. Make security policies consistent: Why are domain controllers all located in a single OU? Answer: To make sure they all get the same GPO Security Settings. When we establish a trust between two Active Directories, we are extending our trust boundary beyond the local forest to our partner. You want to pay careful attention to conflicting settings that would break communications. Here is a classic KB article which describes numerous potential conflicts. Make sure you work together on a common security baseline for both AD Forests. If this is going to be a long term relationship, I recommend a solution in place to identify a naming convention for GPOs with sensitive settings and routinely synchronize or compare these GPOs. Some tools that may help in this effort are the Security Compliance Manager (free!), or Advanced Group Policy Manager (requires MDOP license)
10. Be aware of your partner’s vulnerabilities: When you open your doors to a partner environment, you could be exposing yourself to all the threats they encounter. Here are some questions to ask:
1. How do they enforce system health compliance?
2. How do they deliver patching and security updates?
3. Are they running legacy operating systems?
4. What are their Password Policies?
5. How many Domain Admins do they have, and are any of them service accounts?
6. Have they reported security breaches in the past?
Whenever your users traverse the trust, they could be exposed to credential theft. In turn, their users are now your users. Whatever vulnerabilities and compromises they face, are now your problems. You can set up various roadblocks to prevent these issues.
1. Don’t let privileged accounts logon interactively and restrict what they can access over the network. Stay away from Legacy Operating Systems.
2. Consider Security Considerations for Trusts, implement measures like SID Filtering or Selective Trusts.
3. Secure Privileged Access with a Tier Administration Model and other mitigations.
These are just a few examples of considerations. Just keep in mind that this is a major endeavor and should be well planned. Also, think outside of the box. This may be time to consider hosting services in Azure and merging the two forests in Azure AD, while keeping your on premise nice and isolated.
Randy “Trust me” Turner