Introduction
We recently announced Azure Feature Store which enables customers to use their tools and services to access a feature store in Azure. Feast (Feature store) is an open-source feature store for machine learning projects and helps with productionizing model training and inferencing.
This article walks through the steps to securely deploy Feast in your Azure subscription through Virtual networks(VNet) and to access it from various Azure services like Azure Synapse and Azure Machine Learning.
Installation Steps
Before you begin, you should checkout the repo locally. You can execute these commands in your preferred Linux environment.
NOTE: Since the script requires sudo privileges, Azure Cloudshell is not a recommended environment.
Before you proceed, here are prerequisites to install feast in your Azure subscription.
- An Existing virtual network (VNET) and subnet.
- Azure Kubernetes (AKS) cluster in that VNET.
You can jump to the install feast section if you already have the prerequisite resources.
Here are the commands to create VNET and AKS cluster in that VNET. For more details refer to the documentation here.
# Create a Resource group
sudo az group create --name myakstest --location eastus2
# Create a new VNET
sudo az network vnet create
--resource-group myakstest
--name myAKSVnet
--address-prefixes 192.168.0.0/16
--subnet-name myAKSSubnet
--subnet-prefix 192.168.1.0/24
--location eastus2
# To allow an AKS cluster to interact with other Azure resources, an Azure Active Directory service principal (SP) is used. The service principal needs to have permissions to manage the virtual network and subnet that the AKS nodes use.
# Create a Service principal
sudo az ad sp create-for-rbac --skip-assignment
# Give Network Contributor permissions
VNET_ID=$(sudo az network vnet show --resource-group myakstest --name myAKSVnet --query id -o tsv)
SUBNET_ID=$(sudo az network vnet subnet show --resource-group myakstest --vnet-name myAKSVnet --name myAKSSubnet --query id -o tsv)
sudo az role assignment create --assignee --scope $VNET_ID --role "Network Contributor"
# Finally create the AKS cluster with default networking
sudo az aks create
--resource-group myakstest
--name myAKSCluster
--node-count 3
--network-plugin kubenet
--service-cidr 10.0.0.0/16
--dns-service-ip 10.0.0.10
--pod-cidr 10.244.0.0/16
--docker-bridge-address 172.17.0.1/16
--vnet-subnet-id $SUBNET_ID
--service-principal
--client-secret Steps to install Feast
2. Check the permission on the script by running the following command, the script should have executable permissions.
ls -al ./installfeast.sh Output should look similar to this, r = read, w = write and x= executable
-rwxr-xr-x 1 myuser staff 3385 Oct 6 13:34 installfeast.sh3. If it does not not have executable permissions(x), you can add it by running this command
$ sudo chmod 755 ./installfeast.sh4. Run az login command to make sure you are logged into Azure CLI to execute CLI commands
$ sudo az login5. Before you run the script you need the following information
– Resource Group (-r) : An existing Resource Group Name or a new resource group will be created if it doesn't exist.
– Subscription_id (-s) : Azure Subscription Id. The default subscription will be used if not specified
– AKS Cluster Name (-n) – Name of the existing AKS cluster
– Password (-p): Password to set for the Postgres SQL Database
6. Here is what a sample command looks like
$ sudo ./installfeast.sh -s a6c2a7cc-123-4a1a-b765-xxxxx -r myfeastRG -n myAKSCluster -p NOTE: You might have to run the script as root user depending on your access level.
7. Once your deployment is finished you will see feast services running under the Services and Ingresses option for the AKS cluster
deploy_successAccessing the feast service
From the same virtual network as the AKS cluster
Clients in same VNET can invoke the feast service directly through the Pod IP address as shown below
Use the cluster IP address to run the test script and verify feast installation. To do so you need to set up your feast dev environment first
1. Create Azure Ubuntu VM in same VNET as AKS managed resource group and make sure you can ssh into it by opening up port 22.
2. Install feast-spark in your environment
sudo apt-get update
sudo apt-get -y install python3-pip
pip3 install -r requirements.txt # it is under samples folder3. Edit test_featurestore.py and update the ipaddress to the cluster IP address of internal-feast-core service from the above step.
4. Run the above script using following command
python3 test_featurestore.py5. You should see this output:
Congratulations you have successfully deployed feature store !
From a different virtual network as the AKS cluster
The Feast services running on AKS cluster can be exposed via an internal load balancer and private link service. Clients (Azure VM/ Synapse Notebooks/ Azure ML notebooks) running on separate virtual network can connect to them via private endpoint.
Connecting through Azure Synapse
The Feast services running on AKS cluster can be exposed via an internal load balancer and private link service. Clients (Azure VM/ Synapse Jobs/ Azure ML notebooks) running on separate virtual network can connect to them via private endpoint running in same virtual network as the client. Here is the topology for it
Here are the steps you need to follow to access the feast service through virtual private network.
1. Create AKS Service to expose Feast service components via Internal load balancer
2. Create a private link service to expose the feast service endpoints on ILB
3. Create private endpoint to connect your client to private link service
Here is a simplistic flow diagram of the above topology
Client –> Private Endpoint –> Private Link –> Internal Load Balancer –> Feast Service on AKS
1. Create AKS Service to expose Feast service components via load balancer
Here is the Azure documentation that walks through the steps in more details. Below are the actual commands
– Set the Kubectl config to your cluster
#Select your cluster
az aks get-credentials --resource-group myResourceGroup --name myAKSCluster
#Verify connection
kubectl get nodes– To create an internal load balancer, create two service manifest named internal-lb-feastserving.yaml and internal-lb-feastcore.yaml with the service type LoadBalancer and the azure-load-balancer-internal annotation as shown below:
internal-lb-feastserving.yaml
apiVersion: v1
kind: Service
metadata:
name: internal-feast-serving
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
type: LoadBalancer
ports:
- name: port-1
protocol: TCP
port: 8080
targetPort: 8080
- name: port-2
protocol: TCP
port: 6566
targetPort: 6566
selector:
app: feast-online-serving
component: serving
release: feast-releaseinternal-lb-feastcore.yaml
apiVersion: v1
kind: Service
metadata:
name: internal-feast-core
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
type: LoadBalancer
ports:
- name: port-1
protocol: TCP
port: 8080
targetPort: 8080
- name: port-2
protocol: TCP
port: 6565
targetPort: 6565
selector:
app: feast-core
component: core
release: feast-release– Deploy the services using the kubectl apply and specify the name of your YAML manifest:
kubectl apply -f internal-lb-feastserving.yaml
#Output - service/internal-feast-serving created
kubectl apply -f internal-lb-feastcore.yaml
# Output - service/internal-feast-core created
#Verify the services are created
kubectl get services
# Output
#internal-feast-core LoadBalancer 10.0.34.144 8080:30442/TCP,6565:32265/TCP 4m26s
#internal-feast-serving LoadBalancer 10.0.138.42 8080:30518/TCP,6566:30899/TCP 4m33s
At this point you can go the Managed Resource group for your AKS cluster, you will find an internal load balancer with the name kubernetes-internal.
You can find the managed resource group, under the properties tab of the AKS cluster on the left menu.
Under Front End IPs you will find the two AKS services we just created
If you click on any of them, you will load balancing rules with ports that we defined in .yaml manifest
2. Create private link services to expose the feast service endpoints
Now load balancer is all setup with internal ip address for feast service, we need to create a private link to access it. Here is a detailed document for this, quick instructions below
– On Azure portal, go to Private Link Center and click on Create Private link service
– Provide the name and region, make sure the region is same as the one for your AKS cluster.
– Select the Internal load balancer (kubernetes-internal) for your AKS cluster
– There are two front end IP address – feast-core and feast serving service, pick the feast-core service
– Keep default options for other settings and click Create. Here is the resource view once it is created. Please make a note of the private endpoint connections tab on the left
Repeat the above steps and create a private link for feast-serving service.
3. Create private endpoint to connect your clients to private link service
Some services in Azure provide managed private endpoints, this documentation walks through how you can connect through Synapse workspace to your feast service. For other clients, just create a private endpoint in the same VNET as the client and you will be able to connect to it.
– To create private endpoint in Synapse, you need to create a synapse workspace that has Managed workspace virtual network. To create one, follow this article. To check, if you can use private endpoints with your workspace, go to your Workspace > Manage tab (on left) > Managed Private endpoints should be enabled.
– Hit new button to create private endpoint and select private link service for destination
– Provide a name, select the subscription and the private link service under it. You need to provide a FQDN name to access your service. From Synapse notebooks/jobs you can connect to the service using this FQDN. Make note of this FQDN.
– After creation, the endpoint will still show in provisioning state, at this point you need to go back to Private Link Center.
– You will see the newly create private endpoint, under pending connections, please approve it. You will notice that the provisioning state in Synapse also changes to approved.
Next, repeat the above steps to create another private endpoint for feast-serving service.
Connecting through Azure ML
Here is the topology for connecting to Feast Service on AKS from Azure ML.
For the training notebooks/jobs to connect to Feast Service in a different VNET, the compute instance or compute cluster needs to be in the same VNET as the private endpoint.
For inference notebooks/jobs to connect connect to Feast Service in a different VNET, the inference cluster needs to be in the same VNET as the private endpoint.
You can put the compute instance or compute cluster in a VNET by selecting “Enable Virtual Network” under the Advanced settings tab
Similarly you can put an inference cluster in a VNET from Advanced settings too.
Next, create a private link service for feast service running on AKS's internal load balancer by following the steps above.
Lastly, create a private endpoint in same region as AML compute to connect with private link you just created. AML does not support outbound managed private endpoints, so you will create one by going to Private Link center on Azure portal, you can follow this documentation for actual steps.
You have successfully set up a private connection from your client to Feast Service, next you can run the provided sample here
Next Steps
In this blog we walked through instructions to securely deploy feast and making secure connections to it through various Azure services using private endpoints and VNET. For other detailed examples and functionalities, checkout our Github repository.
Special thanks to Rama Ramani, Xiaochen Wu, Xiaoyong Zhu and Lindsey Allen for their help in creating this blog.