Kubernetes External DNS for Azure DNS & AKS

Introduction

After deploying an application and its services into a cluster, a question rises on the surface, how to access it with a custom domain name ? A simple solution would be to create an A record that points the domain name into the service IP address. This could be done manually, so it will be too hard to scale as you add many services. And this could be fully automated by using External DNS! This tutorial describes how to manage custom domain names in Azure using External in AKS.

External DNS is a controller that watches for new Ingresses and Services with specific annotations, then creates corresponding DNS records in Azure DNS. It is available as an opensource project in Github: https://github.com/kubernetes-sigs/external-dns. It supports more than 30 DNS providers including Azure DNS and Private DNS Zone.

External DNS pods to Azure DNS using one of three methods:

  1. Service principal.
  2. Kubelet Managed Identity.
  3. User assigned Managed Identity controlled by AAD Pod Identity.
Note: Pod Identity is deprecated and will be replaced by Workload Identity. However, ExternalDNS dos not support yet Workload Identity.

Note: If you want to use Kubelet Managed Identity, giving it the Contributor role on the DNS zone is not secure by default. That is because any pod in the cluster can access it. To mitigate this issue, you need to implement a Network Policy that restricts access to the IMDS endpoint to only the ExternalDNS pods.

In this tutorial, you will work with Service Principal.

This article is available as a video in this link: External DNS for Kubernetes.

HoussemDellai_0-1682930879981.png

1. Create an AKS cluster with an ingress controller

Create an AKS cluster.

$AKS_RG="rg-aks-cluster"
$AKS_NAME="aks-cluster"
az group create -n $AKS_RG -l Yousteurope
az aks create -g $AKS_RG -n $AKS_NAME `
              --kubernetes-version "1.25.5" `
              --node-count 3 `
              --network-plugin azure
az aks get-credentials -n $AKS_NAME -g $AKS_RG --overwrite-existing

Install nginx ingress controller to use it later.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx `
     --create-namespace `
     --namespace ingress-nginx `
     --set controller.service.annotations."service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path"=/healthz

2. Create Azure DNS Zone, or use an existing one

You can create a new Azure DNS Zone with or without delegated domain name. Without delegated domain name means it will not be able to publicly resolve the domain name. But you will still see the created DNS records.

In this lab, I use a delegated domain name: houssem.cloud. Replace it with your own.

$DNS_ZONE_NAME="houssem.cloud"
$DNS_ZONE_RG="rg-azure-dns"
az group create -n $DNS_ZONE_RG -l Yousteurope
az network dns zone create -g $DNS_ZONE_RG -n $DNS_ZONE_NAME

3. Create a service principal for ExternalDNS

ExternalDNS will connect to Azure DNS to change its configuration. So, it needs to be authenticated. As mentioned before, You will be using a Service Principal.

$EXTERNALDNS_SPN_NAME="spn-external-dns-aks"
# Create the service principal
$DNS_SPN=$(az ad sp create-for-rbac --name $EXTERNALDNS_SPN_NAME)
$EXTERNALDNS_SPN_APP_ID=$(echo $DNS_SPN | jq -r '.appId')
$EXTERNALDNS_SPN_PASSWORD=$(echo $DNS_SPN | jq -r '.password')

4. Assign the RBAC for the service principal

Grant access to Azure DNS zone for the service principal.

# fetch DNS id and RG used to grant access to the service principal
$DNS_ZONE_ID=$(az network dns zone show -n $DNS_ZONE_NAME -g $DNS_ZONE_RG --query "id" -o tsv)
$DNS_ZONE_RG_ID=$(az group show -g $DNS_ZONE_RG --query "id" -o tsv)
# assign reader to the resource group
az role assignment create --role "Reader" --assignee $EXTERNALDNS_SPN_APP_ID --scope $DNS_ZONE_RG_ID
# assign contributor to DNS Zone itself
az role assignment create --role "DNS Zone Contributor" --assignee $EXTERNALDNS_SPN_APP_ID --scope $DNS_ZONE_ID

Verify role assignments.

az role assignment list --all --assignee $EXTERNALDNS_SPN_APP_ID -o table
# Principal                                         Role                        Scope
# ------------------------------------  --------------------  ----------------------------------------------------------------------------------------------------------------------------------
# 9cc6c0d1-99a3-4d86-9df4-a84df55b8232  Reader                /subscriptions/82f6d75e-85f4-434a-ab74-5dddd9fa8910/resourceGroups/rg-azure-dns
# 9cc6c0d1-99a3-4d86-9df4-a84df55b8232  DNS Zone Contributor  /subscriptions/82f6d75e-85f4-434a-ab74-5dddd9fa8910/resourceGroups/rg-azure-dns/providers/Microsoft.Network/dnszones/houssem.cloud

5. Create a Kubernetes secret for the service principal

ExternalDNS expects to find the Service Principal credentials in a JSON file called azure.json saved as a secret. Let's create the file.

@"
{
  "tenantId": "$(az account show --query tenantId -o tsv)",
  "subscriptionId": "$(az account show --query id -o tsv)",
  "resourceGroup": "$DNS_ZONE_RG",
  "aadClientId": "$EXTERNALDNS_SPN_APP_ID",
  "aadClientSecret": "$EXTERNALDNS_SPN_PASSWORD"
}
"@ > azure.json
cat azure.json
# {
#   "tenantId": "16b3c013-d300-468d-ac64-7eda0820b6d3",
#   "subscriptionId": "82f6d75e-85f4-434a-ab74-5dddd9fa8910",
#   "resourceGroup": "rg-dns-zone-houssem-cloud",
#   "aadClientId": "9cc6c0d1-99a3-4d86-9df4-a84df55b8232",
#   "aadClientSecret": "LJS8Q~ZeuAPJfE7Hjzy6bYZ8NQ4O5YrlJfATxbL6"
# }

Deploy the credentials as a Kubernetes secret.

kubectl create namespace external-dns
# namespace/external-dns created
kubectl create secret generic azure-config-file -n external-dns --from-file azure.json
# secret/azure-config-file created

Verify secret created

kubectl describe secret azure-config-file -n external-dns
# Name:         azure-config-file
# Namespace:    external-dns
# Labels:       
# Annotations:  
# 
# Type:  Opaque
# 
# Data
# ====
# azure.json:  552 bytes

6. Deploy External DNS

ExternalDNS could be deployed through raw YAML manifest, Helm chart or as an operator. For simplicity, you will be using official YAML deployment available here: https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/azure.md#manifest-for-clusters-with-rbac-enabled-cluster-access. Refer to this link to check any possible future change in YAML.

Before deploying the YAML, change the namespace name in ClusterRoleBinding in external-dns.yaml file.

apiVersion: v1
kind: ServiceAccount
metadata:
name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: external-dns
rules:
- apiGroups: [""]
resources: ["services","endpoints","pods", "nodes"]
verbs: ["get","watch","list"]
- apiGroups: ["extensions","networking.k8s.io"]
resources: ["ingresses"]
verbs: ["get","watch","list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: external-dns-viewer
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: external-dns
subjects:
- kind: ServiceAccount
name: external-dns
namespace: external-dns # default
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: external-dns
spec:
strategy:
type: Recreate
selector:
matchLabels:
app: external-dns
template:
metadata:
labels:
app: external-dns
spec:
serviceAccountName: external-dns
containers:
- name: external-dns
image: registry.k8s.io/external-dns/external-dns:v0.13.2
args:
- --source=service
- --source=ingress
- --provider=azure
- --txt-prefix=externaldns-
volumeMounts:
- name: azure-config-file
mountPath: /etc/kubernetes
readOnly: true
volumes:
- name: azure-config-file
secret:
secretName: azure-config-file
kubectl apply -f external-dns.yaml -n external-dns
# serviceaccount/external-dns created
# clusterrole.rbac.authorization.k8s.io/external-dns created
# clusterrolebinding.rbac.authorization.k8s.io/external-dns-vieYour created
# deployment.apps/external-dns created

Note: To deploy ExternalDNS using Helm charts, checkout these resources: https://artifacthub.io/packages/helm/bitnami/external-dns https://github.com/bitnami/charts/tree/main/bitnami/external-dns/#installing-the-chart

Verify deployment.

kubectl get pods,sa -n external-dns
NAME                               READY   STATUS    RESTARTS   AGE
pod/external-dns-5fd5797df-xklxn   1/1     Running   0          96s
NAME                          SECRETS   AGE
serviceaccount/default        0         96m
serviceaccount/external-d

7. Using External DNS with Kubernetes services

You will create a public service of type LoadBalancer. This will create a new public IP address to access the service. Then add an annotation external-dns.alpha.kubernetes.io/hostname with value the custom domain name. This annotation will be red by External DNS to add the IP address to the DNS Zone (in this case app01.houssem.cloud).

You will use this YAML template.

# app-lb.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: app01
spec:
selector:
matchLabels:
app: app01
template:
metadata:
labels:
app: app01
spec:
containers:
- image: mcr.microsoft.com/dotnet/samples:aspnetapp
name: aspnetapp
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: app01-svc
annotations:
external-dns.alpha.kubernetes.io/hostname: app01.houssem.cloud # external-dns configuration
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: app01
type: LoadBalancer
kubectl apply -f app-lb.yaml
# deployment.apps/nginx created
# service/nginx-svc created
kubectl get pods,svc
# NAME                                              READY   STATUS    RESTARTS   AGE
# pod/app01-67745dc95d-bwzgf      1/1        Running   0                 100s
# NAME                       TYPE                 CLUSTER-IP    EXTERNAL-IP    PORT(S)            AGE
# service/app01-svc    LoadBalancer   10.0.95.113     20.86.202.21     80:31067/TCP   100s
# service/kubernetes   ClusterIP          10.0.0.1                       443/TCP            2m30s

Check what is happening in the external DNS pod. Note how External DNS detected the annotation and is creating an A record to the public IP address of the service (20.103.4.205).

kubectl logs external-dns-5fd5797df-xklxn -n external-dns
# time="2023-03-06T09:01:15Z" level=info msg="Updating A record named 'app01' to '20.103.4.205' for Azure DNS zone 'houssem.cloud'."
# time="2023-03-06T09:01:16Z" level=info msg="Updating TXT record named 'externaldns-app01' to '"heritage=external-dns,external-dns/owner=default,external-dns/resource=service/default/app01-svc"' for Azure DNS zone 'houssem.cloud'."
# time="2023-03-06T09:01:16Z" level=info msg="Updating TXT record named 'externaldns-a-app01' to '"heritage=external-dns,external-dns/owner=default,external-dns/resource=service/default/app01-svc"' for Azure DNS zone 'houssem.cloud'."

Check the DNS record is created by external DNS.

az network dns record-set a list -g $DNS_ZONE_RG --zone-name $DNS_ZONE_NAME
# [{
#     "aRecords": [
#       {
#         "ipv4Address": "20.103.57.97"
#       }
#     ],
#     "aaaaRecords": null,
#     "caaRecords": null,
#     "cnameRecord": null,
#     "etag": "99b46f74-8388-44d1-80e9-2aafe1f4802d",
#     "fqdn": "myapp.houssem.cloud.",
#     "id": "/subscriptions/82f6d75e-85f4-434a-ab74-5dddd9fa8910/resourceGroups/rg-dns-zone-houssem-cloud/providers/Microsoft.Network/dnszones/houssem.cloud/A/myapp",
#     "metadata": null,
#     "mxRecords": null,
#     "name": "myapp",
#     "nsRecords": null,
#     "provisioningState": "Succeeded",
#     "ptrRecords": null,
#     "resourceGroup": "rg-dns-zone-houssem-cloud",
#     "soaRecord": null,
#     "srvRecords": null,
#     "targetResource": {
#       "id": null
#     },
#     "ttl": 300,
#     "txtRecords": null,
#     "type": "Microsoft.Network/dnszones/A"
#   }]
app01.png

8. Create a sample app exposed through ingress

You will expose and application through an ingress controller. In the ingress resource you will add a configuration that will be used by External DNS to manage domain names in Azure DNS. That configuration is native to ingress resources which is the host.

# app-ingress.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: app02
spec:
selector:
matchLabels:
app: app02
template:
metadata:
labels:
app: app02
spec:
containers:
- image: mcr.microsoft.com/dotnet/samples:aspnetapp
name: aspnetapp
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: app02-svc
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: app02
type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app02-ingress
spec:
ingressClassName: nginx
rules:
- host: app02.houssemd.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: app02-svc
port:
number: 80
kubectl apply -f app-ingress.yaml
# deployment.apps/app02 created
# service/app02-svc created
# ingress.networking.k8s.io/app02-ingress created
kubectl get pods,svc,ingress
# NAME                                             READY   STATUS     RESTARTS   AGE
# pod/app02-9bdd6845f-vh422       1/1          Running   0                 92s
# NAME                       TYPE            CLUSTER-IP    EXTERNAL-IP    PORT(S)        AGE
# service/app02-svc    ClusterIP      10.0.74.196                 80/TCP          92s
# service/kubernetes   ClusterIP      10.0.0.1                       443/TCP        2m30s
# NAME                                                         CLASS   HOSTS                          ADDRESS        PORTS   AGE
# ingress.networking.k8s.io/app02-ingress   nginx    app02.houssem.cloud   20.73.123.67   80         92s
 

Check the DNS record is created by external DNS

az network dns record-set a list -g $DNS_ZONE_RG --zone-name $DNS_ZONE_NAME
# [
# {
#     "aRecords": [
#       {
#         "ipv4Address": "20.73.123.67"
#       }
#     ],
#     "aaaaRecords": null,
#     "caaRecords": null,
#     "cnameRecord": null,
#     "etag": "f1038e1a-85d3-440e-bd91-fc6f8252e3f1",
#     "fqdn": "app02.houssem.cloud.",
#     "id": "/subscriptions/82f6d75e-85f4-434a-ab74-5dddd9fa8910/resourceGroups/rg-dns-zone-houssem-cloud/providers/Microsoft.Network/dnszones/houssem.cloud/A/app02",
#     "metadata": null,
#     "mxRecords": null,
#     "name": "app02",
#     "nsRecords": null,
#     "provisioningState": "Succeeded",
#     "ptrRecords": null,
#     "resourceGroup": "rg-dns-zone-houssem-cloud",
#     "soaRecord": null,
#     "srvRecords": null,
#     "targetResource": {
#       "id": null
#     },
#     "ttl": 300,
#     "txtRecords": null,
#     "type": "Microsoft.Network/dnszones/A"
#   }
# ]

Let's check the app and DNS resolution. Just open the URL on the browser.

app02.png

Let us check the Azure DNS zone configuration. Note the A records was added with public IP for service and ingress controller.

azure-dns.png

Conclusion

You learned in this tutorial how to configure custom domain names in Azure DNS for external services using External DNS. More details about the project are available here: https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/azure.md.

For exposing custom domain names inside the kubernetes cluster, you can use Core DNS (previously Kube DNS).

Disclaimer
The sample are not supported under any Microsoft standard support program or service. The sample are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.

 

This article was originally published by Microsoft's Core Infrastructure and Security Blog. You can find the original article here.