Quick-Start Guide to Azure Private Endpoints with AKS & Storage

Introduction

Azure Private Endpoints (PE) offer a robust and secure method for establishing connections via a private link. This blog focuses on utilizing PEs to link a Private Azure Service (AKS) cluster with a account, aiming to assist in quick Proof-of-Concept setups. Although we spotlight the service, the insights can be seamlessly applied to other Azure services. We will explore two specific scenarios for setting up Private Endpoints for AKS to interface with Azure :

  1. A Private Endpoint part of the same Virtual Network (VNet) as AKS and in a distinct subnet.
  2. A Private Endpoint in a dedicated VNet, using VNet peering between the AKS and the PE VNet.

Pre-requisites

Private AKS with Private Endpoint Sharing AKS VNet

In this approach, the Private Endpoint (PE) for the Azure Storage Account will be created within the same VNet as the AKS cluster but in a separate subnet. This allows for a simplified network topology while ensuring that the traffic between the AKS cluster and the Storage Account remains private and within the Azure network backbone.

The following diagram depicts the configuration with Storage PE located within the AKS VNet. Connectivity is established via a bastion host linked to a in the AKS VNet or to a VM in a separate VNet that is peered with the AKS VNet. The highlights emphasize the primary focus of this section.varghesejoji_2-1696272561494.png

Setting Up Variables

# Use PowerShell
$resourceGroup="privateep-sharedvnet-rg"
$aksResourceGroup="aks-private" <- Replace with existing>

Resource Group and Storage Account Creation

In this section, we create a resource group and a storage account. The storage account is then configured with a container, and public access is blocked to enhance security.

# Create resource group
az group create --name $resourceGroup --location $region
# Create storage account
az storage account create --name $storageAccountName --resource-group $resourceGroup --location $region --sku Standard_LRS
# Create container in above SA
$storageAccountKey = az storage account keys list --resource-group $resourceGroup --account-name $storageAccountName --query '[0].value' --output tsv
az storage container create --name "container01" --account-name $storageAccountName --account-key $storageAccountKey
# Block public access after container creation
az storage account update --name $storageAccountName --resource-group $resourceGroup --public-network-access Disabled

AKS Configuration

Here, we retrieve the AKS VNet and disable its public FQDN. This ensures that our AKS cluster is private and can only be accessed within the VNet.

# Get Private AKS VNet
$aksMCResourceGroup=az aks show --resource-group $aksResourceGroup --name $aksClusterName --query "nodeResourceGroup" --output tsv
$vnetInfo=az network vnet list --resource-group $aksMCResourceGroup --query '[0].{name:name, id:id}' --output json | ConvertFrom-Json
$aksVnetId=$vnetInfo.id
$aksVnetName=$vnetInfo.name
echo $aksVnetName
# Disable public FQDN on existing AKS Private cluster
az aks update -n $aksClusterName -g $resourceGroup --disable-public-fqdn

Private Endpoint Creation

We create a subnet within the AKS VNet dedicated for the private endpoint. Then, a private endpoint for the storage account is created within this subnet. This allows secure access to the storage account from the AKS cluster for blob storage types.

Create Storage PE subnet on AKS VNet and get PE's Subnet ID

az network vnet subnet create --name $PESubnetName --resource-group $aksMCResourceGroup --vnet-name $aksVnetName --address-prefixes $subnetPrefix --disable-private-endpoint-network-policies true
$PESubnetId = az network vnet subnet show --resource-group $aksMCResourceGroup --vnet-name $aksVnetName --name $PESubnetName --query 'id' --output tsv
echo $PESubnetId

Create Storage PE subnet on AKS VNet and get PE's Subnet ID

az network vnet subnet create --name $PESubnetName --resource-group $aksMCResourceGroup --vnet-name $aksVnetName --address-prefixes $subnetPrefix --disable-private-endpoint-network-policies true
$PESubnetId = az network vnet subnet show --resource-group $aksMCResourceGroup --vnet-name $aksVnetName --name $PESubnetName --query 'id' --output tsv
echo $PESubnetId

Create Storage PE subnet on AKS VNet and get PE's Subnet ID

az network vnet subnet create --name $PESubnetName --resource-group $aksMCResourceGroup --vnet-name $aksVnetName --address-prefixes $subnetPrefix --disable-private-endpoint-network-policies true
$PESubnetId = az network vnet subnet show --resource-group $aksMCResourceGroup --vnet-name $aksVnetName --name $PESubnetName --query 'id' --output tsv
echo $PESubnetId

Create a Private Endpoint for the Storage Account using Subnet ID of Storage PE

# Create Storage PE subnet on AKS VNet and get PE's Subnet ID
az network vnet subnet create --name $PESubnetName --resource-group $aksMCResourceGroup --vnet-name $aksVnetName --address-prefixes $subnetPrefix --disable-private-endpoint-network-policies true
$PESubnetId = az network vnet subnet show --resource-group $aksMCResourceGroup --vnet-name $aksVnetName --name $PESubnetName --query 'id' --output tsv
echo $PESubnetId
# Create a Private Endpoint for the Storage Account using Subnet ID of Storage PE
az network private-endpoint create `
--resource-group $resourceGroup `
--name $StoragePrivateEndpoint `
--vnet-name $aksVnetName `
--subnet $PESubnetId `
--private-connection-resource-id $(az storage account show --name storageAccountName --resource-group $resourceGroup --query "id" --output tsv) `
--group-ids "blob" `
--connection-name "StoragePESharedVNetConnection"

Private DNS Configuration

A private DNS zone is created and a link to the AKS VNet is established. An A-record pointing to the storage private endpoint is added to this DNS zone, enabling name resolution within the VNet.

# Create a Private DNS Zone to associate with Storage PE
az network private-dns zone create --resource-group $resourceGroup --name $PrivateLinkNameSharedVNet
# Link AKS VNet to the Private DNS Zone. Using $aksVnetId, private-link will be in different RG than AKS Vnet (in MC Resource Group)
az network private-dns link vnet create --resource-group $resourceGroup --virtual-network $aksVnetId --name "AksPrivateDnsLink" --zone-name "$PrivateLinkNameSharedVNet" --registration-enabled false
# Get the Private IP Address of the Storage Private Endpoint:
$privateIpAddress = az network private-endpoint show --name $StoragePrivateEndpoint --resource-group $resourceGroup --query 'customDnsConfigs[0].ipAddresses[0]' --output tsv
echo $privateIpAddress
# Add the A-Record of Storage PE to the Private DNS Zone table
az network private-dns record-set a add-record --resource-group $resourceGroup --zone-name $PrivateLinkNameSharedVNet --record-set-name $storageAccountName --ipv4-address $privateIpAddress

Testing and Validation

After setting up the Private Endpoint and configuring the necessary networking components, it is essential to test and validate that the AKS cluster can communicate with the Storage Account over the Private Endpoint.

Basic Connectivity Testing

We deploy a pod within the AKS cluster to test connectivity to the storage account. This verifies that our configurations are correct, and that the AKS cluster can access the storage securely.

# Test to validate AZ CLI access from AKS Pod
$yaml = @"
apiVersion: v1
kind: Pod
metadata:
  name: storage-connectivity-tester
spec:
  containers:
  - name: azure-cli
    image: mcr.microsoft.com/azure-cli
    command:
      - sleep
      - "3600"
"@
$yaml | kubectl apply -f -
# Get shell access to the Pod for interactive command run
kubectl exec storage-connectivity-tester -it -- bash
# Run below in shell to validate NW connectivity to storage accounts from AKS Pod
PrivateLinkNameSharedVNet="privatelink.blob.core.windows.net"
storageAccountName="saprivatesharedvnet"
# Lookup should return PE IP address of Storage FQDN
nslookup $storageAccountName.$PrivateLinkNameSharedVNet
## Test for AZ CLI Storage connectivity
az login
resourceGroup="privateep-sharedvnet-rg"
storageAccountName="saprivatesharedvnet"
storageAccountKey=$(az storage account keys list --resource-group $resourceGroup --account-name $storageAccountName --query '[0].value' --output tsv)
echo $storageAccountKey
# List the containers in the storage account
az storage container list --account-name $storageAccountName --account-key $storageAccountKey

Storage Mount Testing

Deploying a stateful set Pod using below script verifies the ability to mount Azure as a file system using the Blob Container Storage Interface (CSI) driver. This will use the PE to access the storage account.

# Test to validate File mount using Blob CSI driver
$yaml = @"
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: statefulset-blob-nfs
labels:
app: nginx
spec:
serviceName: statefulset-blob-nfs
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: statefulset-blob-nfs
image: mcr.microsoft.com/oss/nginx/nginx:1.19.5
command:
- "/bin/sh"
- "-c"
- while true; do echo $(date) >> /mnt/azureblob/data; sleep 60; done
volumeMounts:
- name: persistent-storage
mountPath: /mnt/azureblob
volumeClaimTemplates:
- metadata:
name: persistent-storage
annotations:
volume.beta.kubernetes.io/storage-class: azureblob-nfs-premium
spec:
accessModes: ["ReadWriteMany"]
resources:
requests:
storage: 100Gi
"@
$yaml | kubectl apply -f -

Validate that the Pod is running and verify from SA that /mnt/azureblob/data has contents
kubectl get ,pv,pvc

Deletion on Completion

# Delete custom resource group on completion to remove all resources created above (other than AKS)
az group delete --name $resourceGroup --yes --no-wait 

Private AKS with Private Endpoint in Dedicated VNet

In this scenario, the Private Endpoint (PE) for the Azure Storage Account will be created in a dedicated VNet separate from the AKS cluster's VNet, making it ideal for hub-spoke configurations.. VNet peering will be established between the AKS VNet and the dedicated VNet to allow private communication between the AKS cluster and the Storage Account.

The following diagram depicts the setup where the Storage Private Endpoint (PE) resides in a VNet that is peered with the AKS VNet. Access to the AKS VNet is facilitated either via a bastion host linked to a VM within the AKS VNet or through a VM in a separate VNet that is peered with the AKS VNet.varghesejoji_3-1696272561499.png

Setting Up Variables

# Use PowerShell
$resourceGroup="privateep-dedicatedvnet-rg"
$aksResourceGroup="aks-private" <- Replace with existing>

Resource Group, Storage Account, and VNet Creation

Like the previous use case, we start by creating a resource group, storage account, and a dedicated VNet for the private endpoint.

# Create resource group
az group create --name $resourceGroup --location $region
# Create storage account
az storage account create --name $storageAccountName --resource-group $resourceGroup --location $region --sku Standard_LRS
# Create container in above SA
$storageAccountKey = az storage account keys list --resource-group $resourceGroup --account-name $storageAccountName --query '[0].value' --output tsv
az storage container create --name "container01" --account-name $storageAccountName --account-key $storageAccountKey
# Block public access after container creation
az storage account update --name $storageAccountName --resource-group $resourceGroup --public-network-access Disabled

Private Endpoint Creation in Dedicated VNet

A subnet is created within the dedicated VNet for the private endpoint. A private endpoint for the storage account is then created within this subnet.

# Create a new PE VNet
az network vnet create --resource-group $resourceGroup --name $PEVNetName --address-prefix $vnetPrefix --location $region
# Create a subnet within the VNet dedicated for the private endpoint
az network vnet subnet create --resource-group $resourceGroup --vnet-name $PEVNetName --name $PESubnetName --address-prefix $subnetPrefix --disable-private-endpoint-network-policies true --disable-private-link-service-network-policies true
$PESubnetId = az network vnet subnet show --resource-group $resourceGroup --vnet-name $PEVNetName --name $PESubnetName --query 'id' --output tsv
echo $PESubnetId
# Create a Private Endpoint for the Storage Account using Subnet ID of Storage PE
az network private-endpoint create `
--resource-group $resourceGroup `
--name $StoragePrivateEndpoint `
--vnet-name $PEVNetName `
--subnet $PESubnetId `
--private-connection-resource-id $(az storage account show --name $storageAccountName --resource-group $resourceGroup --query "id" --output tsv) `
--group-ids "blob" `
--connection-name "StoragePEDedicatedVNetConnection"

Private DNS Configuration in Dedicated VNet

A private DNS zone is created and linked to both the AKS VNet and the dedicated VNet. This ensures name resolution for the storage account across both VNets.

# Create a Private DNS Zone to associate with Storage PE
## Create a new Private DNS Zone for PE
az network private-dns zone create --resource-group $resourceGroup --name $PrivateLinkNameDedicatedVNet
# Link AKS VNet to the Storage Private DNS Zone. Using $aksVnetId, private-link will be in different RG than AKS Vnet (in MC Resource Group)
## Get Private AKS VNet
$aksMCResourceGroup=az aks show --resource-group $aksResourceGroup --name $aksClusterName --query "nodeResourceGroup" --output tsv
$vnetInfo=az network vnet list --resource-group $aksMCResourceGroup --query '[0].{name:name, id:id}' --output json | ConvertFrom-Json
$aksVnetId=$vnetInfo.id
$aksVnetName=$vnetInfo.name
echo $aksVnetName
## Create a VNet link to AKS VNet
az network private-dns link vnet create --resource-group $resourceGroup --virtual-network $aksVnetId --name "AksPrivateDnsLink" --zone-name $PrivateLinkNameDedicatedVNet --registration-enabled false
# Add A-record pointing to Storage PE
## Get the Private IP Address of the Storage Private Endpoint:
$privateIpAddress = az network private-endpoint show --name $StoragePrivateEndpoint --resource-group $resourceGroup --query 'customDnsConfigs[0].ipAddresses[0]' --output tsv
## Add the A-Record of Storage PE to the Private DNS Zone table
az network private-dns record-set a add-record --resource-group $resourceGroup --zone-name $PrivateLinkNameDedicatedVNet --record-set-name $storageAccountName --ipv4-address $privateIpAddress

VNet Peering Configuration

VNet peering is established between the AKS VNet and the dedicated VNet. This allows resources in the AKS VNet to communicate with resources in the dedicated VNet.

# Create VNet peering between AKS and Storage VNets
## Create VNet Peering from AKS VNet to Storage VNet
az network vnet peering create --name peer-aks-storage --resource-group $aksMCResourceGroup --vnet-name $aksVnetName --remote-vnet $(az network vnet show --resource-group $resourceGroup --name $PEVNetName --query id -o tsv) --allow-vnet
# Create VNet Peering from Storage VNet to AKS VNet
az network vnet peering create --name peer-storage-aks --resource-group $resourceGroup --vnet-name $PEVNetName --remote-vnet $(az network vnet show --resource-group $aksMCResourceGroup --name $aksVnetName --query id --out tsv) --allow-vnet-access
# Verification of VNet Peering should say 'Connected'
## Check peering status for AKS VNet
az network vnet peering show --name peer-aks-storage --resource-group $aksMCResourceGroup --vnet-name $aksVnetName --query peeringState
## Check peering status for Storage VNet
az network vnet peering show --name peer-storage-aks --resource-group $resourceGroup --vnet-name $PEVNetName --query peeringState

Testing and Validation in Dedicated VNet

Like the first scenario, after setting up the Private Endpoint in a dedicated VNet and configuring VNet peering, it is crucial to test and validate the connectivity between the AKS cluster and the Storage Account. In a similar manner  we deploy a pod within the AKS cluster to test connectivity to the storage account in the dedicated VNet. Follow the steps in earlier Testing section, with summary listing below.

# Test to validate NW connectivity to storage accounts from AKS Pod
PrivateLinkNameSharedVNet="privatelink.blob.core.windows.net"
storageAccountName="saprivatededicatedvnet"
nslookup $storageAccountName.$PrivateLinkNameSharedVNet
# from Pod CLI run below
az login
resourceGroup="privateep-dedicatedvnet-rg"
storageAccountName="saprivatededicatedvnet"
storageAccountKey=$(az storage account keys list --resource-group $resourceGroup --account-name $storageAccountName --query '[0].value' --output tsv)
echo $storageAccountKey
# List the containers in the storage account
az storage container list --account-name $storageAccountName --account-key $storageAccountKey

Deletion on Completion

# Delete custom resource group on completion to remove all resources created above (other than AKS)
az group delete --name $resourceGroup --yes --no-wait

Conclusion

Azure Private Endpoints provide a secure and private method to connect Azure services like AKS and Storage Accounts. By leveraging Private Endpoints, organizations can ensure that their data remains within the Azure network, reducing the exposure to potential threats. Whether you choose to use a shared VNet or a dedicated VNet for your Private Endpoint, the steps outlined in this blog post will guide you in setting up and validating the connectivity between your AKS cluster and Azure Storage Account.

References

  1. Create a private endpoint by using the Azure CLI
  2. Azure services DNS zone configuration
  3. Connect to a storage account using an Azure Private Endpoint
  4. Connect privately to an Azure container registry using Azure Private Link
  5. Create a private Azure Kubernetes Service (AKS) cluster

Disclaimer
The sample scripts are not supported by any Microsoft standard support program or service. The sample scripts are provided AS IS without a warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.

 

This article was originally published by Microsoft’s System Center Blog. You can find the original article here.