The introduction of managed Kubernetes and serverless solutions overhauled the deployment of workloads in the cloud. Serverless container deployment has taken this one step further. Serverless makes your container hosting options more convenient and agile, with workloads deployed on demand without the work of managing the underlying infrastructure.
Kubernetes remains the most popular container orchestration platform, and all leading cloud service providers have a portfolio of K8s-focused services. These services enable organizations to build and deliver cloud-scale applications through containers, without worrying about running Kubernetes themselves.
Azure Kubernetes Service (AKS) is the managed Kubernetes service offering from Azure. Their platform takes care of everything, from control plane deployment and management, to patching, node provisioning, and more. Customers can bring their containerized applications to Azure or build new applications and deploy them in containers using AKS.
Azure Container Instances (ACI) abstracts the process further by providing a serverless hosting solution for containers. In this article, we’ll explore the features of Azure Container Instances and how it can be used to run serverless pods with AKS.
Customers who don’t need the full orchestration service offered by Kubernetes can utilize Azure Container Instances (ACI) for light-weight, agile container deployments without managing the underlying infrastructure. It’s typically recommended for standalone simple applications, build jobs, or automations that can be executed in isolated containers.
The highlight of ACI is its container isolation, which suits it to hostile multi-tenant deployments. You can specify the CPU and memory requirement for containers so only those resources will be provisioned. Billing is done on a per-second basis, so you only pay for what you use. The service also supports resource-intensive machine learning workloads that require GPU resources (in preview).
ACI supports both Windows and Linux containers and they can be scheduled using the same API. However, some features—such as monitoring, VNet integration, persistent volume mounting, GPU resources (in preview)—are currently only supported for Linux containers. When integrated with Azure VNet, containers deployed in ACI can communicate with other Azure resources in the same VNet or on-premise devices connected through ExpressRoute/VPN.
Azure Kubernetes Service delivers the container orchestration capabilities of K8s preferred by enterprises, while also deploying production-grade applications in the cloud. As a managed service, the K8s master nodes are managed by the platform and customers are required to manage only the agent nodes.
All critical maintenance and patch update activities are handled by Azure, allowing customers to focus on development and delivery of applications. The pricing structure is also flexible, and you pay only for the usage of agent nodes.
AKS is well integrated with supporting Azure services, such as Azure Container Registry, Azure Monitor, Azure AD authentication for RBAC, and more. In addition to Linux, AKS also supports Windows containers. Using Azure AD authentication, customers can use their existing Azure AD users and groups to provide an integrated sign-on experience. Azure Monitor Container Insights service is also integrated out of the box to provide information on container and node health status.
AKS provides different scalability options so you can either scale the number of pods using the pod autoscaler, or increase the number of cluster nodes through cluster autoscaler. However, creating additional nodes to provide the required capacity could take some time and may not be best suited for applications that have quick burst or scaling requirements.
In such scenarios, you can scale AKS using ACI without the overhead of provisioning additional nodes. ACI is used as a secure extension of AKS, using an implementation of Virtual Kubelet called virtual nodes. This enables AKS to deploy pods on demand to ACI in a secure, logical environment, isolated from other users. The cost is also reduced, as you’re only billed for the vCPU and memory used by the deployed pods.
AKS can be scaled using ACI in a few simple steps. We will be using AZ CLI commands in this demo. You can access AZ CLI from Azure Cloud Shell, or install it on your local machine.
$ az provider list --query
user@Azure [ ~ ]$ az provider list --query "[?contains(namespace,'Microsoft.ContainerInstance')]" -o table
Namespace RegistrationState RegistrationPolicy
--------------------------- ------------------- --------------------
Microsoft.ContainerInstance NotRegistered RegistrationRequired
user@Azure [ ~ ]$
In subscriptions where the provider is not registered, you can register it using the following command:
$ az provider register --namespace Microsoft.ContainerInstance
The first step of extending the AKS cluster using serverless ACI is to create the cluster itself, as outlined below:
$ az group create --name demo --location westeurope
The above command creates a resource group named demo in the Azure West Europe region.
$ az network vnet create \
--resource-group demo \
--name AKSnetwork \
--address-prefixes 10.0.0.0/8 \
--subnet-name AKSSubnet \
--subnet-prefix 10.240.0.0/16
This command will deploy a VNet named AKSnetwork in the Resource group created in step 1. The command defines the address space of the VNet as 10.0.0.0/8, and it creates a subnet called AKSsubnet with a subnet prefix 10.240.0.0/16.
$ az ad sp create-for-rbac --skip-assignment
user@Azure [ ~ ]$ az ad sp create-for-rbac --skip-assignment
Option '--skip-assignment' has been deprecated and will be removed in a future release.
The output includes credentials that you must protect. Be sure that you do not include these credentials in your code or check the credentials into your source control. For more information, see https://aka.ms/azadsp-cli
{
"appId": "acea04e6-9d91-434c-90a1-2143b6ba7f61",
"displayName": "azure-cli-2022-10-10-14-22-11",
"password": "<COPY THIS PASSWORD>",
"tenant": "3ef364a9-c74c-42b8-a07e-a46e9d449fd1"
}
Record the appId and password. You’ll need them during the AKS cluster deployment.
$ az network vnet show --resource-group demo --name AKSnetwork --query id -o tsv
user@Azure [ ~ ]$ az network vnet show --resource-group demo --name AKSnetwork --query id -o tsv
/subscriptions/331fbdae-2630-495d-a6a0-beaee1c25ca9/resourceGroups/demo/providers/Microsoft.Network/virtualNetworks/AKSnetwork
Assign contributor rights in the VNet for the AKS service principle, replace <vnetid> with the output of the previous command, and replace <appid> with the output from command in step 3.
$ az role assignment create --assignee <appId> --scope <vnetId> --role Contributor
user@Azure [ ~ ]$ az role assignment create --assignee acea04e6-9d91-434c-90a1-2143b6ba7f61 --scope /subscriptions/331fbdae-2630-495d-a6a0-beaee1c25ca9/resourceGroups/demo/providers/Microsoft.Network/virtualNetworks/AKSnetwork --role Contributor
{
"canDelegate": null,
"condition": null,
"conditionVersion": null,
"description": null,
"id": "/subscriptions/331fbdae-2630-495d-a6a0-beaee1c25ca9/resourceGroups/demo/providers/Microsoft.Network/virtualNetworks/AKSnetwork/providers/Microsoft.Authorization/roleAssignments/1dbcb02c-306f-43bd-83c6-c78efc9586ce",
"name": "1dbcb02c-306f-43bd-83c6-c78efc9586ce",
"principalId": "b2fb59a5-27fd-4a9c-8c81-bf7a452345d2",
"principalType": "ServicePrincipal",
"resourceGroup": "demo",
"roleDefinitionId": "/subscriptions/331fbdae-2630-495d-a6a0-beaee1c25ca9/providers/Microsoft.Authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c",
"scope": "/subscriptions/331fbdae-2630-495d-a6a0-beaee1c25ca9/resourceGroups/demo/providers/Microsoft.Network/virtualNetworks/AKSnetwork",
"type": "Microsoft.Authorization/roleAssignments"
}
$ az network vnet subnet show --resource-group demo --vnet-name AKSnetwork --name AKSSubnet --query id -o tsv
user@Azure [ ~ ]$ az network vnet subnet show --resource-group demo --vnet-name AKSnetwork --name AKSSubnet --query id -o tsv
/subscriptions/331fbdae-2630-495d-a6a0-beaee1c25ca9/resourceGroups/demo/providers/Microsoft.Network/virtualNetworks/AKSnetwork/subnets/AKSSubnet
<subnetId> , <appId> / <password> with values obtained from step 5 and 3 respectively.$ az aks create \
--resource-group demo \
--name AKSCluster1 \
--node-count 1 \
--network-plugin azure \
--service-cidr 10.1.0.0/16 \
--dns-service-ip 10.1.0.10 \
--docker-bridge-address 172.17.0.1/16 \
--vnet-subnet-id <subnetId> \
--service-principal <appId> \
--client-secret <password> \
--generate-ssh-keys
$ az network vnet subnet create \
--resource-group demo \
--vnet-name AKSnetwork \
--name VirtualNodeSubnet \
--address-prefixes 10.241.0.0/16
$ az aks enable-addons \
--resource-group demo \
--name AKSCluster1 \
--addons virtual-node \
--subnet-name VirtualNodeSubnet
$ az aks get-credentials --resource-group demo --name AKSCluster1
$kubectl get nodes
user@Azure [ ~ ]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-nodepool1-25388420-vmss000000 Ready agent 7m28s v1.23.12
virtual-node-aci-linux Ready agent 2m39s v1.19.10-vk-azure-aci-v1.4.5-dev
You can see that ACI is listed in the output as one of the AKS nodes, indicated by the name “virtual-node-aci-linux.” This means the AKS cluster has been successfully extended using ACI.
apiVersion: v1
kind: Service
metadata:
name: my-service
namespace: default
labels:
app: my-demoapp
spec:
selector:
app: my-demoapp
ports:
- protocol: TCP
port: 80
targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-deployment
spec:
replicas: 1
selector:
matchLabels:
app: my-demoapp
template:
metadata:
labels:
app: my-demoapp
spec:
containers:
- name: nginx
image: public.ecr.aws/z9d2n7e1/nginx:1.19.5
ports:
- containerPort: 80
nodeSelector:
kubernetes.io/role: agent
beta.kubernetes.io/os: linux
type: virtual-kubelet
tolerations:
- key: virtual-kubelet.io/provider
operator: Exists
- key: azure.com/aci
effect: NoSchedule
The nodeSelector configuration highlighted above specifies that the workload has to run on the virtual nodes using ACI.
kubectl apply -f sample-aks.yaml
$kubectl get pods
user@Azure [ ~ ]$ kubectl get pods
NAME READY STATUS RESTARTS AGE
my-deployment-8d48dc6cf-dgggn 1/1 Running 0 52s
user@Azure [ ~ ]$
$kubectl describe pod my-deployment-5c654ff6f-s2v8j
user@Azure [ ~ ]$ kubectl describe pod my-deployment-8d48dc6cf-dgggn
Name: my-deployment-8d48dc6cf-dgggn
Namespace: default
Priority: 0
Service Account: default
Node: virtual-node-aci-linux/
Labels: app=my-demoapp
pod-template-hash=8d48dc6cf
Annotations: <none>
Status: Running
IP: 10.140.0.9
IPs:
IP: 10.140.0.9
Controlled By: ReplicaSet/my-deployment-8d48dc6cf
Containers:
nginx:
Container ID: aci://040cafa3ad93c8e0c2a4a711c6649c577c4219900da5cfb78f1c0c1964799939
Image: public.ecr.aws/z9d2n7e1/nginx:1.19.5
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Running
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-d22h4 (ro)
Volumes:
kube-api-access-d22h4:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: beta.kubernetes.io/os=linux
kubernetes.io/role=agent
type=virtual-kubelet
Tolerations: azure.com/aci:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
virtual-kubelet.io/provider op=Exists
You should now see that the node is deployed to ACI.
As you’ve seen, you can easily extend Azure AKS clusters using ACI. This ensures your compute resources are consumed on demand without the overhead of maintaining the infrastructure. As the pods deployed to ACI run in isolated environments, it can even be used in multi-tenant use cases that need enhanced security.
Using AKS and ACI, customers can easily address burst scenarios for Kubernetes pods without the additional cost for backend nodes. This also saves you time, as pods can be deployed in a matter of seconds rather than the additional time required to deploy AKS nodes.
In short, running serverless containers using AKS and ACI enables enhanced security, right sizing of resources, and reduced cost for running containerized workloads in Azure.