Private AKS and ACR Using Private Endpoint – Part 2/2

Introduction

The objective of this tutorial is to create a private environment for AKS, ACR and a JumpBox VM. AKS and ACR will be accessible only through Private Endpoint.

As of part 1, our did connect securely through Private Endpoint to the private AKS . In this next part, we'll cover the remaining steps:

  • Configure access to ACR using Private Endpoint.
  • Setup connection between the VM and ACR.
  • Setup connection between the AKS and ACR.

At the end of the tutorial, we should have the following architecture implemented for AKS, ACR and VM. Storage Account and Key Vault are just for illustration as we still can use Private Endpoint to access them privately.

HoussemDellai_0-1644146201986.jpeg

Part 1 is available here: Private AKS and ACR Using Private Endpoint – Part 1/2 (microsoft.com)

This tutorial is also available as a video.

1. Configure access to ACR using Private Endpoint

In the previous tutorial, we had already created an ACR attached to AKS. By default, ACR is public. Here want to disable its public FQDN and make it accessible only using Private Endpoint (aka PE). Private Endpoint will create a private IP address in a Subnet/VNET. Documentation about private ACR is here.

Private Endpoint will disable NSGs on its host Subnet. We don't want this to happen to AKS Subnet, so we can create another Subnet in AKS VNET dedicated for Private Endpoints. Let's do that from the Azure portal.

HoussemDellai_1-1644146202010.png

Note: It is also possible to create a dedicated VNET for PEs in scenarios where we use Hub and Spoke model.

We can have a Spoke for PEs peered only with the Spokes that needs access to the PEs resources.

Now let's got to create the Private Endpoint for ACR. Private Endpoint is a feature of the Premium plan of ACR.

HoussemDellai_2-1644146202021.png

Go to Overview then Update and update the plan to Premium.

HoussemDellai_3-1644146202024.png

Go to Networking, you can see the more options now with the premium plan. Set Public network access to Disabled to disable public access to the ACR through the internet.

Our goal here is to connect to the ACR (from the DevBox VM and AKS) using the private endpoint.

Using the private endpoint will allow only resources from our network to access ACR.

HoussemDellai_4-1644146202032.png

Then select Private access and create private endpoint connection:

HoussemDellai_5-1644146202040.png

Let's follow the 5 steps to create the private endpoint for ACR. First, we choose the resource group where to create the Private Endpoint and its Network Interface Card (), here I choose the AKS node resource group.

HoussemDellai_6-1644146202048.png

 In the second step, we keep the default configuration.

HoussemDellai_7-1644146202057.png

In configuration section, we choose the VNET and Subnet where to attach the Network Interface for the PE. Here I choose the AKS VNET, but I choose a dedicated subnet (the one created earlier). Note the information about disabling NSGs on this subnet. 

Then we enable Integrate with private DNS zone. This will create a Private DNS Zone and add a link to the VNET. This will allow resources deployed in that VNET to be able to resolve the private endpoint link.

I choose to deploy the Private DNS Zone into the AKS node resource group as I've done with the PE.

HoussemDellai_8-1644146202087.png

Finally, we create the PE.

HoussemDellai_9-1644146202105.png

We go into the AKS nodes resource group, and we will see the Private Endpoint, card and the Private DNS Zone.

HoussemDellai_10-1644146202142.png

Check the config of the Private DNS Zone. Note the private IP addresses (10.241.0.4 for the Storage Account behind ACR and 10.241.0.5 for ACR).

HoussemDellai_11-1644146202156.png

Also note the private link: acr1privateaks.privatelink.azurecr.io

HoussemDellai_12-1644146202161.png

Note the link added to the AKS VNET. This means each resource in this VNET can connect to the ACR through the PE. Which also means our AKS can connect to it to pull the images.

HoussemDellai_13-1644146202169.png

Let's also check the Private Endpoint configuration. Note how it relies on the Private DNS Zone to resolve the ACR link.

HoussemDellai_14-1644146202201.png

Note: The card is in read-only mode, and it follows the same life cycle as the PE.

2. Setup connection between the VM and ACR

Our objective now is to connect to private ACR from the VM and push an image. We will use a DevBox/JumpBox VM to push the image. If this VM is in the same AKS VNET, it will be able to connect directly to ACR. But in this demo, we will use a VM hosted in its own VNET.

In real life this is the case for architectures where a JumpBox or a DevOps build agent in one VNET are used to deploy to resources in other VNETs.

For the DevBox VM to connect to the private ACR through the Private Endpoint sitting in another VNET, we need to do 2 steps:

  • Create a VNET Peering between VM VNET and ACR Private Endpoint VNET.
  • Add link to the JumpBox VM's VNET in the ACR Private DNS Zone.

Let's go to one of the two VNETs and create a VNET Peering between VM VNET and ACR Private Endpoint VNET. But this was already done in the previous tutorial! Just in case, check the created peering:

HoussemDellai_15-1644146202209.png

Let's add link to the JumpBox VM's VNET in the ACR Private DNS Zone. In the ACR Private DNS Zone, add a link to the JumpBox VM's VNET.

HoussemDellai_16-1644146202216.png

Now we are ready to connect from the VM to private ACR. Take a moment to install Azure CLI, Docker and kubectl in the VM.

We go to the ACR to import an image from MCR using the DevBox VM.

$ $acrName=”acr1privateaks”
$ az acr login -n $acrName
Login Succeeded
$ az acr import -n $acrName –source mcr.microsoft.com/oss/nginx/nginx:1.21.4 –image nginx:1.21.4

 Alternatively, we can also pull an image from MCR, tag it, and push it to ACR.

$ docker pull mcr.microsoft.com/hello-world
$ docker tag mcr.microsoft.com/hello-world “$acrName.azurecr.io/hello-world:v1.0”
$ docker push “$acrName.azurecr.io/hello-world:v1.0”

HoussemDellai_18-1644146202230.png

Great! Now the DevBox VM connects to private ACR and pushes the images.

3. Setup connection between AKS and private ACR

To enable the connection between AKS and private ACR:

  • AKS should have access to ACR Private Endpoint (in our case ACR's Private Endpoint is in the same AKS VNET).
  • Add a link to AKS VNET in ACR's Private DNS zone.

For AKS to connect to Private ACR: just add the AKS VNET as a Link to the Private DNS Zone of the private ACR.

HoussemDellai_19-1644146202247.png

From the private DNS Zone, we can see the two links.

HoussemDellai_20-1644146202258.png

Now, we can validate that private AKS connects successfully to private ACR. Let's deploy a container from ACR and watch how that deploys successfully.

$ kubectl run nginx –image=”$acrName.azurecr.io/nginx:1.21.4″
$ kubectl get pods -w

HoussemDellai_21-1644146202264.png

Conclusion and final notes

We had learned in this tutorial provision a secure AKS environment using Private Endpoint. The only allowed connections will be through Bastion using the DevBox VM. AKS and ACR both are private and accessible only from the private VNET network using Private Endpoint.

Typically, with AKS, we would also use Storage Account, Key Vault, . We can make these resources private and accessible only through Private Endpoint.

It is possible to disable the public FQDN of AKS API Server. Details here:

https://docs.microsoft.com/en-us/azure/aks/private-clusters#disable-public-fqdn

To resume, as it is mentioned briefly here, virtual network peering is one way to access your private . To use virtual network peering, you need to set up a link between virtual network and the private DNS zone.

  1. Go to the node resource group in the Azure portal.
  2. Select the private DNS zone.
  3. In the left pane, select the Virtual network link.
  4. Create a new link to add the virtual network of the VM to the private DNS zone. It takes a few minutes for the DNS zone link to become available.
  5. In the Azure portal, navigate to the resource group that contains your cluster's virtual network.
  6. In the right pane, select the virtual network. The virtual network name is in the form aks-vnet-*.
  7. In the left pane, select Peerings.
  8. Select Add, add the virtual network of the VM, and then create the peering.
  9. Go to the virtual network where you have the VM, select Peerings, select the AKS virtual network, and then create the peering. If the address ranges on the AKS virtual network and the VM's virtual network clash, peering fails.

An important note from this source: Multiple private endpoints can be created using the same private link resource. For a single network using a common DNS server configuration, the recommended practice is to use a single private endpoint for a given private link resource. Use this practice to avoid duplicate entries or conflicts in DNS resolution.

Another important note from this source, if you plan on using private endpoints to access resources from an on-premises system, deploy the endpoints in your hub. With this arrangement, you can take advantage of some of the benefits:

    • Using network security groups to control access to your resources
    • Managing your private endpoints in a centralized location

If you're planning on accessing resources from applications that you've deployed in Azure, the situation's different:

    • If only one application needs access to your resources, deploy Private Endpoint in that application's spoke.
    • If more than one application needs access to your resources, deploy Private Endpoint in your hub.

Disclaimer

The sample scripts are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.

 

This article was originally published by Microsoft's Secure Blog. You can find the original article here.