Kentaro is CEO and Solutions Architect at Coder Society. With his in-depth knowledge of software development and cloud technologies, Kentaro often takes on the lead engineer's role. His analytical, organized, and people-oriented nature makes him an apt advisor on software projects and flexible staffing.
Try our new React app builder today
Try it for freeUse Terraform to Create and Manage a HA AKS Kubernetes Cluster in Azure
Learn how to use Terraform to manage a highly-available Azure AKS Kubernetes cluster with Azure AD integration and Calico network policies enabled.
What is Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS) is a managed Kubernetes offering in Azure which lets you quickly deploy a production ready Kubernetes cluster. It allows customers to focus on application development and deployment, rather than the nitty gritties of Kubernetes cluster management. The cluster control plane is deployed and managed by Microsoft while the node and node pools where the applications are deployed, are handled by the customer.
The AKS cluster deployment can be fully automated using Terraform. Terraform enables you to safely and predictably create, change, and improve infrastructure. It also supports advanced AKS configurations, such as availability zones, Azure AD integration, and network policies for Kubernetes.
AKS features in focus
Let’s take a look at the key AKS features we’ll be covering in this article.
AKS deployment across multilpe availability zones
Ensuring high availability of deployments is a must for enterprise workloads. Azure availability zones protect resources from data center-level failures by distributing them across one or more data centers in an Azure region.
AKS clusters can also be deployed in availability zones, in which the nodes are deployed across different zones in a region. In case of a data center failure, the workloads deployed in the cluster would continue to run from nodes in a different zone, thereby protecting them from such incidents.
Azure Active Directory integration
With identity considered the new security perimeter, customers are now opting to use Azure AD for authentication and authorization of cloud-native deployments. AKS clusters can be integrated with Azure Active Directory so that users can be granted access to namespaces in the cluster or cluster-level resources using their existing Azure AD credentials. This eliminates the need for multiple credentials when deploying and managing workloads in an AKS cluster.
This is of even greater benefit in hybrid cloud deployments, in which on-premises AD credentials are synced to Azure AD. It delivers a consistent, unified experience for authentication and authorization. Figure 1 below shows this high-level AKS authentication flow when integrated with Azure Active Directory.
Pod traffic control through network policy implementation
By default, all pods in an AKS cluster can communicate with each other without any restrictions. However, in production, customers would want to restrict this traffic for security reasons. This can be achieved by implementing network policies in a Kubernetes cluster. Network policies can be used to define a set of rules that allow or deny traffic between pods based on matching labels.
AKS supports two types of network implementations: Kubenet (basic networking) and Azure CNI (advanced networking). Customers can also choose between two types of network policies: Azure (native) or Calico network policies (open source). While Azure network policies are supported only in Azure CNI, Calico is supported in both Kubenet- and Azure CNI-based network implementations.
Deployment prerequisites
Following are the prerequisites for the deployment of the AKS cluster:
Azure subscription access: It is recommended that users with contributor rights run the Terraform scripts. During deployment, an additional resource group is created for the AKS nodes. Restricted permissions may lead to deployment failures.
Azure AD server and client application: OpenID Connect is used to integrate Azure Active Directory with the AKS cluster. Two Azure AD applications are required to enable this: a server application and a client application. The server application serves as the endpoint for identity requests, while the client application is used for authentication when users try to access the AKS cluster via the kubectl command. Microsoft offers a step-by-step guide for creating these Azure AD applications.
Terraform usage from Cloud Shell: Azure Cloud Shell has Terraform installed by default in the bash environment. You can use your favorite text editor like vim or use the code editor in Azure Cloud Shell to write the Terraform templates. Refer to Microsoft’s guide to get started with Terraform in Azure Cloud Shell.
Creating a Terraform template
To create the templates, Terraform uses HashiCorp Configuration Language (HCL), as it is designed to be both machine friendly and human readable. For a more in-depth understanding of Terraform syntax, refer to the Terraform documentation. The values that change across deployments can be defined as variables and are either provided through a variables file or during runtime when the Terraform templates are applied.
In this section, we’ll describe the relevant modules of the Terraform template to be used to create the cluster.
Note: The Terraform template as well as the variable and output files for this deployment are all available in the GitHub repository.
Network setup
The following block of Terraform code should be used to create the Azure VNet and subnet, which are required for the Azure CNI network implementation:
resource "azurerm_virtual_network" "demo" {name = "${var.prefix}-network"location = azurerm_resource_group.demo.locationresource_group_name = azurerm_resource_group.demo.nameaddress_space = ["10.1.0.0/16"]}resource "azurerm_subnet" "demo" {name = "${var.prefix}-akssubnet"virtual_network_name = azurerm_virtual_network.demo.nameresource_group_name = azurerm_resource_group.demo.nameaddress_prefixes = ["10.1.0.0/22"]}
var.prefix: A prefix will be defined in the Terraform variable files which is used to differentiate the deployment.
demo: This is the local name which is used by Terraform to reference the defined resources (e.g. Azure VNet and subnet). It can be renamed to suit your use case.
address_space and address_prefixes: This refers to the address space for the VNet and subnet. You can replace the values with your preferred private IP blocks.
Azure AD integration
To enable the Azure AD integration we need to provide the server application, client application, and Azure AD tenant details. The following code block should be used in the AKS cluster definition to enable RBAC for the AKS cluster and to use Azure AD for RBAC authentication.
role_based_access_control {azure_active_directory {client_app_id = var.client_app_idserver_app_id = var.server_app_idserver_app_secret = var.server_app_secrettenant_id = var.tenant_id}enabled = true}
var.client_app_id: This variable refers to the client app ID of the Azure AD client application which was mentioned in the prerequisites section.
var.server_app_id: This variable refers to the server app ID of the Azure AD server application which was mentioned in the prerequisites section.
var.server_app_secret: This variable refers to the secret created for the Azure AD server application.
var.tenant_id: This variable refers to the Azure AD tenant ID associated with the subscription where the cluster will be deployed. This value can be obtained from the Azure portal or through the Azure CLI.
Network policy configuration
The following Terraform code will be used in the AKS cluster definition to enable Calico network policies. Note that this can be configured only during cluster deployment and any changes will require a recreation of the cluster.
network_profile {network_plugin = "azure"load_balancer_sku = "standard"network_policy = "calico"}
network_plugin: The value should be set to azure to use CNI networking.
load_balancer_sku: The value should be set to standard, as we will be using virtual machine scale sets.
network_policy: The value should be set to calico since we’ll be using Calico network policies.
Node pool and availability zone configuration
The following code will be used to configure the node pools and availability zone.
default_node_pool {name = "default"node_count = 2vm_size = "Standard_D2_v2"type = "VirtualMachineScaleSets"availability_zones = ["1", "2"]enable_auto_scaling = truemin_count = 2max_count = 4# Required for advanced networkingvnet_subnet_id = azurerm_subnet.demo.id}
node_count: This refers to the initial amount of nodes to be deployed in the node pool.
vm_size: Standard_D2_v2 is used in this sample; it can be replaced with your preferred SKU.
type: This should be set to VirtualMachineScaleSets so that the VMs can be distributed across availability zones.
availability_zones: Lists the available zones to be used.
enable_auto_scaling: This should be set to true to enable autoscaling.
The variables min_count and max_count should be set to define the minimum and maximum node count within the node pool. The value here should be between 1 and 100.
Deploying the HA AKS cluster
Download the Terraform files from the GitHub repository to your Cloud Shell session and edit the configuration parameters in accordance with your AKS cluster deployment requirements. The guidance provided in the previous section can be used to update these values.
- Run the following commands to clone the GitHub repository in CloudShell:
git clone https://github.com/coder-society/terraform-aks-azure.gitCloning into 'terraform-aks-azure'...remote: Enumerating objects: 12, done.remote: Counting objects: 100% (12/12), done.remote: Compressing objects: 100% (10/10), done.remote: Total 12 (delta 1), reused 12 (delta 1), pack-reused 0Unpacking objects: 100% (12/12), done.Checking connectivity... done.
- Go into the
/terraformdirectory and run theterraform initcommand to initialize Terraform:
terraform initInitializing the backend...Initializing provider plugins...- Finding hashicorp/azurerm versions matching "~> 2.0"...- Installing hashicorp/azurerm v2.28.0...- Installed hashicorp/azurerm v2.28.0 (signed by HashiCorp)Terraform has been successfully initialized!You may now begin working with Terraform. Try running "terraform plan" to seeany changes that are required for your infrastructure. All Terraform commandsshould now work.If you ever set or change modules or backend configuration for Terraform,rerun this command to reinitialize your working directory. If you forget, othercommands will detect it and remind you to do so if necessary.
- Export the Terraform variables to be used during runtime, replace the placeholders with environment-specific values. You can also define the values in the variables file.
export TF_VAR_prefix=<Environment prefix>export TF_VAR_client_app_id=<The client app ID of the AKS client application>export TF_VAR_server_app_id=<The server app ID of the AKS server application>export TF_VAR_server_app_secret=<The secret created for AKS server application>export TF_VAR_tenant_id=<The Azure AD tenant id>
- Create the Terraform plan by executing
terraform plan -out out.plan.
terraform plan -out out.planRefreshing Terraform state in-memory prior to plan...The refreshed state will be used to calculate this plan, but will not bepersisted to local or remote state storage.------------------------------------------------------------------------An execution plan has been generated and is shown below.Resource actions are indicated with the following symbols:+ createTerraform will perform the following actions:# azurerm_kubernetes_cluster.demo will be created+ resource "azurerm_kubernetes_cluster" "demo" {+ dns_prefix = "cs-aks"+ fqdn = (known after apply)+ id = (known after apply)+ kube_admin_config = (known after apply)+ kube_admin_config_raw = (sensitive value)+ kube_config = (known after apply)+ kube_config_raw = (sensitive value)+ kubelet_identity = (known after apply)+ kubernetes_version = (known after apply)+ location = "westeurope"+ name = "cs-aks"+ node_resource_group = (known after apply)+ private_cluster_enabled = (known after apply)+ private_fqdn = (known after apply)+ private_link_enabled = (known after apply)+ resource_group_name = "cs-rg"+ sku_tier = "Free"+ tags = {+ "Environment" = "Development"}+ addon_profile {+ aci_connector_linux {+ enabled = (known after apply)+ subnet_name = (known after apply)}+ azure_policy {+ enabled = (known after apply)}+ http_application_routing {+ enabled = (known after apply)+ http_application_routing_zone_name = (known after apply)}+ kube_dashboard {+ enabled = (known after apply)}+ oms_agent {+ enabled = (known after apply)+ log_analytics_workspace_id = (known after apply)+ oms_agent_identity = (known after apply)}}+ auto_scaler_profile {+ balance_similar_node_groups = (known after apply)+ max_graceful_termination_sec = (known after apply)+ scale_down_delay_after_add = (known after apply)+ scale_down_delay_after_delete = (known after apply)+ scale_down_delay_after_failure = (known after apply)+ scale_down_unneeded = (known after apply)+ scale_down_unready = (known after apply)+ scale_down_utilization_threshold = (known after apply)+ scan_interval = (known after apply)}+ default_node_pool {+ availability_zones = [+ "1",+ "2",]+ enable_auto_scaling = true+ max_count = 4+ max_pods = (known after apply)+ min_count = 2+ name = "default"+ node_count = 2+ orchestrator_version = (known after apply)+ os_disk_size_gb = (known after apply)+ type = "VirtualMachineScaleSets"+ vm_size = "Standard_D2_v2"+ vnet_subnet_id = (known after apply)}+ identity {+ principal_id = (known after apply)+ tenant_id = (known after apply)+ type = "SystemAssigned"}+ network_profile {+ dns_service_ip = (known after apply)+ docker_bridge_cidr = (known after apply)+ load_balancer_sku = "standard"+ network_plugin = "azure"+ network_policy = "calico"+ outbound_type = "loadBalancer"+ pod_cidr = (known after apply)+ service_cidr = (known after apply)+ load_balancer_profile {+ effective_outbound_ips = (known after apply)+ idle_timeout_in_minutes = (known after apply)+ managed_outbound_ip_count = (known after apply)+ outbound_ip_address_ids = (known after apply)+ outbound_ip_prefix_ids = (known after apply)+ outbound_ports_allocated = (known after apply)}}+ role_based_access_control {+ enabled = true+ azure_active_directory {+ client_app_id = "f9bf8772-aaba-4773-a815-784b31f9ab8b"+ server_app_id = "fa7775b3-ea31-4e99-92f5-8ed0bac3e6a8"+ server_app_secret = (sensitive value)+ tenant_id = "8f55a88a-7752-4e10-9bbb-e847ae93911d"}}+ windows_profile {+ admin_password = (sensitive value)+ admin_username = (known after apply)}}# azurerm_resource_group.demo will be created+ resource "azurerm_resource_group" "demo" {+ id = (known after apply)+ location = "westeurope"+ name = "cs-rg"}# azurerm_subnet.demo will be created+ resource "azurerm_subnet" "demo" {+ address_prefix = (known after apply)+ address_prefixes = [+ "10.1.0.0/22",]+ enforce_private_link_endpoint_network_policies = false+ enforce_private_link_service_network_policies = false+ id = (known after apply)+ name = "cs-subnet"+ resource_group_name = "cs-rg"+ virtual_network_name = "cs-network"}# azurerm_virtual_network.demo will be created+ resource "azurerm_virtual_network" "demo" {+ address_space = [+ "10.1.0.0/16",]+ guid = (known after apply)+ id = (known after apply)+ location = "westeurope"+ name = "cs-network"+ resource_group_name = "cs-rg"+ subnet = (known after apply)}Plan: 4 to add, 0 to change, 0 to destroy.------------------------------------------------------------------------This plan was saved to: out.planTo perform exactly these actions, run the following command to apply:terraform apply "out.plan"
- Use the
terraform apply out.plancommand to apply the plan.
Once successfully deployed, the details of the cluster, network, etc. will be shown in the command line:
terraform apply out.planazurerm_resource_group.demo: Creating...azurerm_resource_group.demo: Creation complete after 1s [id=/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourceGroups/cs-rg]azurerm_virtual_network.demo: Creating...azurerm_virtual_network.demo: Still creating... [10s elapsed]azurerm_virtual_network.demo: Creation complete after 15s [id=/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourceGroups/cs-rg/providers/Microsoft.Network/virtualNetworks/cs-network]azurerm_subnet.demo: Creating...azurerm_subnet.demo: Creation complete after 2s [id=/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourceGroups/cs-rg/providers/Microsoft.Network/virtualNetworks/cs-network/subnets/cs-subnet]azurerm_kubernetes_cluster.demo: Creating...azurerm_kubernetes_cluster.demo: Still creating... [10s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [20s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [30s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [40s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [50s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [1m0s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [1m10s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [1m20s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [1m30s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [1m40s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [1m50s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [2m0s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [2m10s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [2m20s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [2m30s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [2m40s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [2m50s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [3m0s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [3m10s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [3m20s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [3m30s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [3m40s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [3m50s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [4m0s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [4m10s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [4m20s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [4m30s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [4m40s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [4m50s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [5m0s elapsed]azurerm_kubernetes_cluster.demo: Still creating... [5m10s elapsed]azurerm_kubernetes_cluster.demo: Creation complete after 5m16s [id=/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourcegroups/cs-rg/providers/Microsoft.ContainerService/managedClusters/cs-aks]Apply complete! Resources: 4 added, 0 changed, 0 destroyed.The state of your infrastructure has been saved to the pathbelow. This state is required to modify and destroy yourinfrastructure, so keep it safe. To inspect the complete stateuse the `terraform show` command.State path: terraform.tfstateOutputs:client_certificate =kube_config = apiVersion: v1clusters:- cluster:certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUV5akNDQXJLZ0F3SUJBZ0lSQU01eXFucWJNNmoxekFhbk8vczdIeVV3RFFZSktvWklodmNOQVFFTEJRQXcKRFRFTE1Ba0dBMVVFQXhNQ1kyRXdJQmNOTWpBd09USXlNakEwTWpJeFdoZ1BNakExTURBNU1qSXlNRFV5TWpGYQpNQTB4Q3pBSkJnTlZCQU1UQW1OaE1JSUNJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBZzhBTUlJQ0NnS0NBZ0VBCnc5WlYwd0hORkJiaURUcnZMQ1hJU2JoTHdScFZ1WVowMVJ5QzU0QWo1WUJjNzZyRXZIQVZlRXhWYlRYc0wrdDEKMVJPOG85bVY2UG01UXprUzdPb2JvNEV6ampTY2RBcjUvazJ4eC9BLzk0ZFl2Q20wWHZQaktXSWxiS08yaDBuNQp4VW03OFNsUks4RFVReUtGWmxTMEdXQ0hKbEhnNnpVZU8xMVB5MTQvaXBic1JYdTNFYVVLeWFJczgrekxSK2NNCm5tSU5wcHFPNUVNSzZKZVZKM0V0SWFDaFBNbHJiMVEzNHlZbWNKeVkyQmZaVkVvV2pRL1BMNFNFbGw0T0dxQjgKK3Rlc3I2TDBvOW5LT25LVlB0ZCtvSXllWnQ1QzBiMnJScnFDU1IyR09VZmsvMTV3emFiNTJ6M0JjempuV0VNOApnWWszZlBDU3JvNGE5a0xQVS9Udnk1UnZaUjJsc09mUWk3eGZpNm91dzJQeEkxc1ZPcmJnRWdza2o5Qmc4WnJYCk5GZjJpWlptSFM0djNDM1I4Q25NaHNRSVdiSmlDalhDclZCak1QbzVNS0xzNEF5U1M2dU1MelFtSjhNQWVZQlEKSHJrWEhZa21OeHlGMkhqSVlTcTdjZWFOVHJ1dTh2SFlOT2c3MGM5aGEvakZ0MXczWVl4N3NwbGRSRGpmZHZiQgpaeEtwbWNkUzY3RVNYT0dtTEhwZis1TTZXMVI3UWQwYk1SOGRQdFZJb1NmU2RZSFFLM0FDdUxrd1ZxOWpGMXlnCiswcklWMC9rN0F6bzNnUlVxeFBIV0twcHN2bFhaZCtsK0VqcTRLMnFMRXd4MlFOMDJHL1dGVUhFdGJEUXAvZWYKZGxod3Z0OHp1VklIbXE0ejlsMGZSOU9QaGN6UFpXR0dyWnUrTlQ2cm5RTUNBd0VBQWFNak1DRXdEZ1lEVlIwUApBUUgvQkFRREFnS2tNQThHQTFVZEV3RUIvd1FGTUFNQkFmOHdEUVlKS29aSWh2Y05BUUVMQlFBRGdnSUJBQjQ0CmRzbExhRzFrSzFHaFZiR1VEbWtWNndsTHJ1a1F3SHc2Vkt2azJ0T0UwMU1Hbng5SkJwVFFIL2FMdWYxUTN0WVAKaVhWSnp1bGsxOWg2MjUrcEs0ekpQcDZSWkhMV3htK2M0a1pLejlwV3F3RERuNmZqTVlKOEdMNVZFOUwvQUdSRgpscEI5ZTZNOVNGY20ra2lMVVlLazg3VG1YaGd4dzFXYXhtaDEwd01DNlZPOGNuZlVRQkJJQkVMVXhNZy9DRE9SCjZjSGFTdU5ETlg0UG80eFF3NnV4c0d1R2xDbHltOUt4Z2pIYjQ5SWp2NnN5clRIcGkrVkxmQ3d4TmdLZUwxZ1cKNURVR3ZoMmpoTVJqNDhyV3FoY3JjUHI4aWtoeVlwOXMwcFJYQVlUTk52SlFzaDhmYllOcTIzbDZwVW16dUV0UwpQS2J2WUJDVmxxa29ualZiRkxIeXJuRWNFbllqdXhwYy94bWYvT09keHhUZzlPaEtFQTRLRTQySFJ2cW1SZER5CkFldVhIcUxvUm54TXF1Z0JxL0tTclM2S0tjQW11eVJWdkhJL21MUlhmY1k1VThCWDBXcUF0N1lrWm54d1JnRkQKQndRcnEvdDJrUkMySSsxR1pUd2d1Y3hyc0VrYlVoVG5DaStVbjNDRXpTbmg5anBtdDBDcklYaDYzeC9LY014egpGM0ZXNWlnZDR4MHNxYk5oK3B4K1VSVUlsUmZlMUxDRWg3dGlraVRHRGtGT05EQXBSQUMycnUrM0I5TlpsR0hZCm9jWS9tcTlpdUtXTUpobjFHeXJzWGZLZXQrakliZzhUNzZzaEora0E4djU3VmdBdlRRSEh1YTg2SHl6d1d2Z0QKQ2ZaZFhpeURvZGlWRXhPNGlGaG00T1dhZld5U0ltSUsrOCs1Z2daZwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==server: https://cs-aks-f9e8be99.hcp.westeurope.azmk8s.io:443name: cs-akscontexts:- context:cluster: cs-aksuser: clusterUser_cs-rg_cs-aksname: cs-akscurrent-context: cs-akskind: Configpreferences: {}users:- name: clusterUser_cs-rg_cs-aksuser:auth-provider:config:apiserver-id: fa7775b3-ea31-4e99-92f5-8ed0bac3e6a8client-id: f9bf8772-aaba-4773-a815-784b31f9ab8bconfig-mode: "1"environment: AzurePublicCloudtenant-id: 8f55a88a-7752-4e10-9bbb-e847ae93911dname: azure
- Browse to the resource pool in the Azure portal to view the cluster and the network which was created by the deployment:
- Retrieve the admin kubeconfig using the Azure cli:
az aks get-credentials --resource-group $prefix-rg --name $prefix-aks --admin --overwrite-existing
- Run the following command to list the nodes and availability zone configuration:
kubectl describe nodes | grep -e "Name:" -e "failure-domain.beta.kubernetes.io/zone"Name: aks-default-36042037-vmss000000failure-domain.beta.kubernetes.io/zone=westeurope-1Name: aks-default-36042037-vmss000001failure-domain.beta.kubernetes.io/zone=westeurope-2
failure-domain.beta.kubernetes.io/zone is a label associated with Kubernetes nodes that indicates the zone in which it is deployed. The output shows that the nodes are deployed across two availability zones in Western Europe.
Configure the Azure Active Directory integration
- Create a group in Azure AD.
GROUP_ID=$(az ad group create --display-name dev --mail-nickname dev --query objectId -o tsv)
- Retrieve the resource ID of the AKS cluster
AKS_ID=$(az aks show \--resource-group $prefix-rg \--name $prefix-aks \--query id -o tsv)
- Create an Azure role assignment so that any member of the
devgroup can usekubectlto interact with the Kubernetes cluster.
az role assignment create \--assignee $GROUP_ID \--role "Azure Kubernetes Service Cluster User Role" \--scope $AKS_ID
- Add yourself to the
devAD group.
USER_ID=$(az ad signed-in-user show --query objectId -o tsv)az ad group member add --group dev --member-id $USER_ID
With the admin kubeconfig, create a
developmentandproductionKubernetes namespace.kubectl create namespace development kubectl create namespace productionReplace the
groupObjectIdwith the resource ID of the previously created group and apply therolebinding.yamlfile.
sed -i '' "s/groupObjectId/$GROUP_ID/g" rolebinding.yamlkubectl apply -f rolebinding.yaml
- Run the following command to get the cluster credentials before testing Azure AD integration.
az aks get-credentials --resource-group $prefix-rg --name $prefix-aks --overwrite-existing
- Run the following kubectl command to see the Azure AD integration in action:
kubectl get pods --namespace developmentTo sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code DSFV9H98W to authenticate.No resources found in development namespace.
Enter the code in the device login page followed by your Azure AD login credentials:
Note that only users in the dev group will be able to log in through this process.
- Try to access resources in the
productionnamespace:
kubectl get pods --namespace productionError from server (Forbidden): pods is forbidden: User "kentaro@codersociety.com" cannot list resource "pods" in API group "" in the namespace "production"
Configure network policies with Calico
- To test Calico network policy, create an httpbin service and deployment in a namespace using the k8s/httpbin.yaml.
kubectl apply -f httpbin.yaml --namespace development
- Create a network policy which restricts all inbound access to the deployment using k8s/networkpolicy.yaml. We only allow network access from pods with the label
app: webapp.
kubectl apply -f networkpolicy.yaml --namespace development
- Create a new pod and test access to the httpbin service. From the command prompt of the pod, try to access the httpbin service over port 8000. The access will timeout. You can type “exit” to exit and delete the pod after testing.
kubectl run --rm -it --image=alpine frontend --namespace developmentIf you don't see a command prompt, try pressing enter./ # wget --timeout=2 http://httpbin:8000Connecting to httpbin:8000 (10.0.233.179:8000)wget: download timed out/ # exit
- Create a new test pod, but this time with labels matching the ingress rules. Then run the wget command to check access to httpbin service over port 8000.
kubectl run --rm -it --image=alpine frontend --labels app=webapp --namespace developmentIf you don't see a command prompt, try pressing enter./ # wget --timeout=2 http://httpbin:8000Connecting to httpbin:8000 (10.0.233.179:8000)saving to 'index.html'index.html 100% |************************************************************************************| 9593 0:00:00 ETA'index.html' saved
You can see that it's now possible to retrieve the index.html which shows that the pod can access the httpbin service, since the pod labels match the ingress policy.
Remove demo resources
Go into the terraform directory and run terraform destroy. You get asked if you really want to delete the resources where you confirm by entering yes.
terraform destroyazurerm_resource_group.demo: Refreshing state... [id=/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourceGroups/cs-rg]azurerm_virtual_network.demo: Refreshing state... [id=/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourceGroups/cs-rg/providers/Microsoft.Network/virtualNetworks/cs-network]azurerm_subnet.demo: Refreshing state... [id=/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourceGroups/cs-rg/providers/Microsoft.Network/virtualNetworks/cs-network/subnets/cs-subnet]azurerm_kubernetes_cluster.demo: Refreshing state... [id=/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourcegroups/cs-rg/providers/Microsoft.ContainerService/managedClusters/cs-aks]An execution plan has been generated and is shown below.Resource actions are indicated with the following symbols:- destroyTerraform will perform the following actions:# azurerm_kubernetes_cluster.demo will be destroyed- resource "azurerm_kubernetes_cluster" "demo" {- api_server_authorized_ip_ranges = [] -> null- dns_prefix = "cs-aks" -> null- enable_pod_security_policy = false -> null- fqdn = "cs-aks-f9e8be99.hcp.westeurope.azmk8s.io" -> null- id = "/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourcegroups/cs-rg/providers/Microsoft.ContainerService/managedClusters/cs-aks" -> null- kube_admin_config = [- {- client_certificate = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUUvVENDQXVXZ0F3SUJBZ0lSQUxFazBXdFZWb1dFS0Nra21aeGFaRkl3RFFZSktvWklodmNOQVFFTEJRQXcKRFRFTE1Ba0dBMVVFQXhNQ1kyRXdIaGNOTWpBd09USXlNakEwTWpJeFdoY05Nakl3T1RJeU1qQTFNakl4V2pBdwpNUmN3RlFZRFZRUUtFdzV6ZVhOMFpXMDZiV0Z6ZEdWeWN6RVZNQk1HQTFVRUF4TU1iV0Z6ZEdWeVkyeHBaVzUwCk1JSUNJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBZzhBTUlJQ0NnS0NBZ0VBd0xnVWRpZjJ2ZVFraUdXaDVNbS8KUUdNSzJ2MFMxcDFJKzBQTmRVWVNZSko0dWVGNFpQVUZFcEJyMm9WR2txU29QNUIrNHRlY2RTVkgrL1FvaWI2RQpQYVJwTUNrYnhBSFZPZ1RTcGdJWkliQlp3WjRGamJHbXRtS0lSV1RyR25lcUZSOFFMUHlGdG5TODlNVktUdEU2CjZyOWc0ODRJVTJaM3Q1Wlc4UTdHdFBnU2p4VWQrYWtkTHJZMVUyNzU3TEQyZXBsWlA4UVU3bTRJQ3pXWDFQWWIKMTFTQjJyQjhMc1hpYWRQS2gyQW1tV2t2Y2JkVzFrQW5zWnJ3OHQ2elZIbytlUk5OWWpLdHNXczJ4TXFvdVduVQpJR0UwcjRCaDhXbTFDanluSnNGTXk5S056c1FGV3IzM0hieWU1b00zQU1YN0VaQ1JxRlpLWjhaa2NWbTFaaXdTCi9hNjlJYkVTbmYrbGszbkh4QzJFQjdoVTlQc1FvYkFPUU91MUprbWZMaGsxYTF4N1B2Y0lXbm0rTnAzdko1dlQKMk9mcW1uLzJ3VGFwMkUwSlVpWHFjV3h6YVN6bEpBbXJVdkt3TXZZcWtHVmdRdHk4OGZUM0J4NmFVWUxwQXFVRQpXZG1kWGhFN1BaWXlnT1pFWHIvUVJkSW5BcWZLNmFiWEduc3h2QVFPYVFMWTlBRHk3NkNWem9CamhpdHh5bjFzCm4rU3VQK3l4Y3I3Tmp2VUtHK2g2UzlzMm56eDd5Wm9rUENMSXF4Sm5xdTU4UzhkM1lPR0cvTmVTTll2aGhmNkMKVjFWdEdHaWFsTGFqUGNCd0h1cTFuR0U1WEkvaXlWQk5pdGtmMWk5alMrNnFvU2VsbUJyMUV3YmI1OWlvekUxRApXRnloQWZWNWQ3MEx4QnBheDYrc1M5OENBd0VBQWFNMU1ETXdEZ1lEVlIwUEFRSC9CQVFEQWdXZ01CTUdBMVVkCkpRUU1NQW9HQ0NzR0FRVUZCd01DTUF3R0ExVWRFd0VCL3dRQ01BQXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnSUIKQUs4UWJLUFZyL3QwVlRncS8vSG12UmplKzlYVUNUaUgxRXpDTkFRTVVkcFpjcXJWYXhIMlF3ZVM2SVkrRGU3ZApBUUhYMTM1M0JUc3I0L0VNazVaUTJIaUdjMFZCRzRKSE1NYmNkcjRWb0EwdjhiUmxJSFZRQ2E1QWZhOUFRQTFYCjgvT0pFMUVLeWtFU21jQThkQnA0YTh5cGcwbkZFQzNPQlFlcWx1MjFFK2swU3NKT1VScHU3WE4wUVVWV2NnSFcKNFNOWWtzV2JmRkN6ekpCWmthTmdRUnlhZDJVYWNTQ0REM1ZiNWVHYTljTmpYMzgvbkdZUFhQNlQzbzZFQkJnMApxM0ZZaW9TN0lPZ0xuVSt3cld5b2hXeGNyM2ZUK0J5MW5UOG9oeVVFNDVONm4wMldwclVlLzJGUU9ERjZUOWcvCkkxemhWOVlJbW5wcDMvY1BrZldKYjFFK0hTMU04V284dUdCa25xaVpJVzFaM1NJVFVReVlqWUJkY2grNnVSTWgKMEdxakRHNXViZU1sU0pONkNSUHBoMVpzOERLSjN2MjFUdkYwTjJaL3UyTHU2TGdkaWZLWUZvbStmME0vVUJFUQpRNjVsVHhNeUs5MXZzNDRaMWQ3ODNxcG5ab2RaUWo5VTBqWGVtWnZyMFRtWlh2UHhSdHByTWpXaVNDZVZWNjdSCjFldGQ3NWJiMmFldUF1V2VmYVZscmorc0dRUU1IN1JuUUh1WXhOaktNKzRxU2Z3eHhyeXQ0Q0VUcThFT1grRlcKOFllTEsxTlErOXRaTXZTQ1NwdmRZUnV2NlUvdHVDUnZZTUVLMnMwN1NtdjRDZWFqU25hbW53S0JZZUZld0dNNQpIL0VkSVRwekRQQjVoQkFWeEVlb0czU3FENHo4anpQS1daVWpXY3pTbDZTbwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="- client_key = "LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlKS1FJQkFBS0NBZ0VBd0xnVWRpZjJ2ZVFraUdXaDVNbS9RR01LMnYwUzFwMUkrMFBOZFVZU1lKSjR1ZUY0ClpQVUZFcEJyMm9WR2txU29QNUIrNHRlY2RTVkgrL1FvaWI2RVBhUnBNQ2tieEFIVk9nVFNwZ0laSWJCWndaNEYKamJHbXRtS0lSV1RyR25lcUZSOFFMUHlGdG5TODlNVktUdEU2NnI5ZzQ4NElVMlozdDVaVzhRN0d0UGdTanhVZAorYWtkTHJZMVUyNzU3TEQyZXBsWlA4UVU3bTRJQ3pXWDFQWWIxMVNCMnJCOExzWGlhZFBLaDJBbW1Xa3ZjYmRXCjFrQW5zWnJ3OHQ2elZIbytlUk5OWWpLdHNXczJ4TXFvdVduVUlHRTByNEJoOFdtMUNqeW5Kc0ZNeTlLTnpzUUYKV3IzM0hieWU1b00zQU1YN0VaQ1JxRlpLWjhaa2NWbTFaaXdTL2E2OUliRVNuZitsazNuSHhDMkVCN2hVOVBzUQpvYkFPUU91MUprbWZMaGsxYTF4N1B2Y0lXbm0rTnAzdko1dlQyT2ZxbW4vMndUYXAyRTBKVWlYcWNXeHphU3psCkpBbXJVdkt3TXZZcWtHVmdRdHk4OGZUM0J4NmFVWUxwQXFVRVdkbWRYaEU3UFpZeWdPWkVYci9RUmRJbkFxZksKNmFiWEduc3h2QVFPYVFMWTlBRHk3NkNWem9CamhpdHh5bjFzbitTdVAreXhjcjdOanZVS0craDZTOXMybnp4Nwp5Wm9rUENMSXF4Sm5xdTU4UzhkM1lPR0cvTmVTTll2aGhmNkNWMVZ0R0dpYWxMYWpQY0J3SHVxMW5HRTVYSS9pCnlWQk5pdGtmMWk5alMrNnFvU2VsbUJyMUV3YmI1OWlvekUxRFdGeWhBZlY1ZDcwTHhCcGF4NitzUzk4Q0F3RUEKQVFLQ0FnQnZBaG1YTGRIczg1c3ZqZ3RBOUF6Y0U3RFBEM05vZDlUd0Z0QWtPeWFleGdBUVloV3RZWTE0Y2dRTwpMVExIaVZ6NHNFekdjWmZIeXArNk81dVdMRTJVREQ0aTVhcitybWVhTWVqOGdyempNT2VpcFZsaGt2RUtvWnNKCkRlWjJxbk1vRTJxSDN6Vk9NZFFkMGY3Smc2L0NSRmFWSWJxZC82bjU3L2xJaFZCa01YalBQa1N6Nkh2TXlsdlIKSVYySXZ5NWExRFlhaXVIYnJUbW82MGYzL1lOdjkxZU5GcGVSZ1o2M2dxMW9hVFFTcmdvTUlLVSthRm4wN2VEWQpwUHI3TUNjSUt0d3FNakxtdlhFZ3pmTitTYjFNb1hGdG5pL01sUzBaSm5MSjJoSllYWUlkbGIvWDB4Q2k2bUZGCk9sUFdlRFAwbkNlcXBYbmFhT2EyZkF3SFBGLzdEQzlRUkRDNjNTZ2w3YVkxSDMzVyt0Q2JNYjQzMTZLMEJWbVoKaUpNZDduM3FNU0hQc0FPeVl4UDNrazM1RHBCdlBvc1RkbG13cGxHK0psdHY1eDAxdWU4L3lGRWtRTExITkNiVgpiaTBndTlndExpRGtwZ0M4RFo4eWljNTRzTS9oaHNFUTExY0FQT2xNRlJDTzJGZ2cyNFRrVWthZkxHb210TWk4CktDWnJpUWM2VjVVZWdITnVpeWJKN0lUeVhuNGJvU2piTExHZTNGZVlQUWQ5S2VBam03T0hsVXVTemVBZDZrY3cKaG9zV2ViWlB3ZzlNaEVuUllicmxWWEZyREJEWE9lVW12elRGOGc1Q09xK243M3JLSTNlZzRTNTk1eWpYRm1RTwpTRTd3Nk52bnlEQ1RvbDU1THY4MytkaUFNbzlqR0haSzdyRndGcGZMN0J5dG1qVGJzUUtDQVFFQTUzUlhJR1JBCnd6WTR2UWR2WjdnVkc4Z2tqWTNhNTV6UEdsR2dabGtqdWxVVUZyMmJtWWZXMlFudTVsQ2t2RURjQThxSUdOdXcKck1QNHFxSldNM3pZTVQvS05LajhnOHR1ZDhOQVNaK3FCQ0p1WlZBYlJaNGlaQXZYVmVZRTZhYXU5MjB6NVoxZQpVdTZFVmQxeC9XNit2RFp5ZU5XbFIvZmc1MG44K2tiTExteG1Kc2haeEZxRHI1eTNGeEdXK1ZqS2grTjA5Rm5OClhLWkx2KzdXMU1heG1sNkNrbzVtUVZvbW9GU28zTlVNeTRXRU9GREllTlN6bkZ4ZE9WKzVGc2k5UzFOSnZjRWsKZ2tiMUtVL0x5ZHJ1ditmNVZqOTcvOG9Rcm5tWFE5NGhQbW1veTJTckZ3Z0ZiWmE3M0l0S1N5N2I5bjdIa3RJRQpFZ3J2ZlpUcml2L1ZNd0tDQVFFQTFTZ2c4MktmS29uanc4b2VTdDZVZDRhdkJSdUdYREdlcndYZGRsdm1oc2VMCkUxNDdKaVJ5Um9rYTRGMkNMWkI2NGtySURnUkZxaDdxZjJLNVFTQk1QWXFhRTQ3WDlzZGlqZ0lpMUR6SThSVm4KejlyWGxtdzcycjhWN084Q2RPY1ZySEtuSXVXVkJSd2h5clJsbGM1L1h1eG1oaGZwRmN5d3ZEUlgyKzdsbUV0aQpMSlFKVm56NGY0MkdnbGFHWEt5SmwyaGtRdDE4NWZ0WGhUengrell3bGZHWi9qbjlvUE9Qa3JpZU1yb2xqbzJoClBjNktmd2hEZTlKNTA4R3FNNEJ6MEczQjQ0RkNDMWhIMHd4SnY2aE81RFkzL3FGQ25hQ1U0SStlUjBQVzZXY0MKZ1RsRGxMeUtwd0czZzBzcENySCtrREpLYVBqUGVhS1FsTmtianJTV3BRS0NBUUVBbHV3SHUvbGpPV2RyeStiRApRQkNLd3hqWXJPem81c29iU1lBY1pXQ09xWHU4bzY5emZNTlUxeVZoQUJGcHVjOVpKNmV5NHZLdDI1blYxZjRRCjAzWCt5dTViZmNjTEVTMWZsUHhlT1NQQml2eWdtN09HZFBqT1dBcFltWXhwZTZuU3dVZ1Y1UTJlYWRsWnRWdTIKYnBqK0NtQStlSWhuUSt4Z1hMQ2tJdFp5dW95NGQyV0JFMFlxUkNLZVNJNlJzWG15WnJWc2w4RE81akVSaDgvSAppZXNkK0JqVWI1Z25HVW9ka2NKaWNjMENrTnM1QWplNjRQOWhOdjRMVTlRVkxzUXFtcWx1bGlzUkVWb1BscWFQCnJjbnlrSFJFNDNaMTlxN2QvY2NQV1pQSWZaZ01Gc1JIdzdiWlEwSmNzVXlxWHlmcENteFUybW5UZWFoanpiR0QKZlptZ2ZRS0NBUUFIc3RaVjFBY0pvMGROcC93bUdobmtvMEdvL3BDQXZlNE1SanIwYm1kS0VPVHVBeVpCdjJrOQpNUEIra0FJR29VUSs3aEtCcHhmWkNCclNGUCs1NFcrL2ZVVUpWY3hwQmxTQjZvUFZoSWlCWkpPR1IxSW9CYXEzCndOVUs1S3NEQytHVmcrS1RlUlZEeFB0WGRlS0JZWjdxRDhHNE1CN2tBYXVVY0pPSHh2NFYzUXNqcndrVFRab3cKQ1MyRmdaaUN1bHlSMGx4a3FkazcrVEwxQmZsN2FENmkrOEhqRTdjY1hBK2diZmlRdm5aaXlxeTdMYjJFendpWQo3VVluSnNSOTdiTEJJV1d5VU5YUTBSUnZBKytaODNzOTlOTmE1L29lOVZETE40U3c4RHRQM0wrVGFUME9ueXltCjBZSU9ST1dybERnc2Z4Uis3QldhUUF2V3hHeWhYOVpkQW9JQkFRQ3E1ZmdIZ3VsdU5wSU5Pc045cFRQL2JTYXgKRjVaR1FRQXEyZTNnSW42SEUzRkYzWnlmVGJESkRPbHNIMktoQzlVZ1daT01YUVU1d1N0RVlDZ25ZZkw2azBVQgpnZi95aVh6NXd6bW1CREkzNU1hY25McW1XeVJKS3N2aWpEY0VjTmZ2V1JORlVKK2RvVEFoQ1VnY2dRbTRzWm9PClloMHJKay9CenFvQXBETzhiOGpFMDI2T2dRd0tVUlBlekhUcjdLb0pyckQyWTlDQ0RFd29UdjFQUDI5dDNJcWUKbVdYMjJWcTJqWGdmSDJLcVBVdWk1VGtQZlphM29zR29RaysrQlFFaGxTQXhMKzdSRnFkbjduYkRVQzYvUmM4MgowdlFKS3BiVnlRVDRnbUtEd1BhazNGZGdvQkFHQ3J3NTh5ekRacEFTQXVrZzRpbzVNRHJGbERNR2dSZ1IKLS0tLS1FTkQgUlNBIFBSSVZBVEUgS0VZLS0tLS0K"- cluster_ca_certificate = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUV5akNDQXJLZ0F3SUJBZ0lSQU01eXFucWJNNmoxekFhbk8vczdIeVV3RFFZSktvWklodmNOQVFFTEJRQXcKRFRFTE1Ba0dBMVVFQXhNQ1kyRXdJQmNOTWpBd09USXlNakEwTWpJeFdoZ1BNakExTURBNU1qSXlNRFV5TWpGYQpNQTB4Q3pBSkJnTlZCQU1UQW1OaE1JSUNJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBZzhBTUlJQ0NnS0NBZ0VBCnc5WlYwd0hORkJiaURUcnZMQ1hJU2JoTHdScFZ1WVowMVJ5QzU0QWo1WUJjNzZyRXZIQVZlRXhWYlRYc0wrdDEKMVJPOG85bVY2UG01UXprUzdPb2JvNEV6ampTY2RBcjUvazJ4eC9BLzk0ZFl2Q20wWHZQaktXSWxiS08yaDBuNQp4VW03OFNsUks4RFVReUtGWmxTMEdXQ0hKbEhnNnpVZU8xMVB5MTQvaXBic1JYdTNFYVVLeWFJczgrekxSK2NNCm5tSU5wcHFPNUVNSzZKZVZKM0V0SWFDaFBNbHJiMVEzNHlZbWNKeVkyQmZaVkVvV2pRL1BMNFNFbGw0T0dxQjgKK3Rlc3I2TDBvOW5LT25LVlB0ZCtvSXllWnQ1QzBiMnJScnFDU1IyR09VZmsvMTV3emFiNTJ6M0JjempuV0VNOApnWWszZlBDU3JvNGE5a0xQVS9Udnk1UnZaUjJsc09mUWk3eGZpNm91dzJQeEkxc1ZPcmJnRWdza2o5Qmc4WnJYCk5GZjJpWlptSFM0djNDM1I4Q25NaHNRSVdiSmlDalhDclZCak1QbzVNS0xzNEF5U1M2dU1MelFtSjhNQWVZQlEKSHJrWEhZa21OeHlGMkhqSVlTcTdjZWFOVHJ1dTh2SFlOT2c3MGM5aGEvakZ0MXczWVl4N3NwbGRSRGpmZHZiQgpaeEtwbWNkUzY3RVNYT0dtTEhwZis1TTZXMVI3UWQwYk1SOGRQdFZJb1NmU2RZSFFLM0FDdUxrd1ZxOWpGMXlnCiswcklWMC9rN0F6bzNnUlVxeFBIV0twcHN2bFhaZCtsK0VqcTRLMnFMRXd4MlFOMDJHL1dGVUhFdGJEUXAvZWYKZGxod3Z0OHp1VklIbXE0ejlsMGZSOU9QaGN6UFpXR0dyWnUrTlQ2cm5RTUNBd0VBQWFNak1DRXdEZ1lEVlIwUApBUUgvQkFRREFnS2tNQThHQTFVZEV3RUIvd1FGTUFNQkFmOHdEUVlKS29aSWh2Y05BUUVMQlFBRGdnSUJBQjQ0CmRzbExhRzFrSzFHaFZiR1VEbWtWNndsTHJ1a1F3SHc2Vkt2azJ0T0UwMU1Hbng5SkJwVFFIL2FMdWYxUTN0WVAKaVhWSnp1bGsxOWg2MjUrcEs0ekpQcDZSWkhMV3htK2M0a1pLejlwV3F3RERuNmZqTVlKOEdMNVZFOUwvQUdSRgpscEI5ZTZNOVNGY20ra2lMVVlLazg3VG1YaGd4dzFXYXhtaDEwd01DNlZPOGNuZlVRQkJJQkVMVXhNZy9DRE9SCjZjSGFTdU5ETlg0UG80eFF3NnV4c0d1R2xDbHltOUt4Z2pIYjQ5SWp2NnN5clRIcGkrVkxmQ3d4TmdLZUwxZ1cKNURVR3ZoMmpoTVJqNDhyV3FoY3JjUHI4aWtoeVlwOXMwcFJYQVlUTk52SlFzaDhmYllOcTIzbDZwVW16dUV0UwpQS2J2WUJDVmxxa29ualZiRkxIeXJuRWNFbllqdXhwYy94bWYvT09keHhUZzlPaEtFQTRLRTQySFJ2cW1SZER5CkFldVhIcUxvUm54TXF1Z0JxL0tTclM2S0tjQW11eVJWdkhJL21MUlhmY1k1VThCWDBXcUF0N1lrWm54d1JnRkQKQndRcnEvdDJrUkMySSsxR1pUd2d1Y3hyc0VrYlVoVG5DaStVbjNDRXpTbmg5anBtdDBDcklYaDYzeC9LY014egpGM0ZXNWlnZDR4MHNxYk5oK3B4K1VSVUlsUmZlMUxDRWg3dGlraVRHRGtGT05EQXBSQUMycnUrM0I5TlpsR0hZCm9jWS9tcTlpdUtXTUpobjFHeXJzWGZLZXQrakliZzhUNzZzaEora0E4djU3VmdBdlRRSEh1YTg2SHl6d1d2Z0QKQ2ZaZFhpeURvZGlWRXhPNGlGaG00T1dhZld5U0ltSUsrOCs1Z2daZwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="- host = "https://cs-aks-f9e8be99.hcp.westeurope.azmk8s.io:443"- password = "15f169a920129ead802a0de7c5be9500abf964051850b652acf411ab96e587c4e9a9255b155dc56225245f84bcacfab5682d74b60bb097716fca8a14431e8c5e"- username = "clusterAdmin_cs-rg_cs-aks"},] -> null- kube_admin_config_raw = (sensitive value)- kube_config = [- {- client_certificate = ""- client_key = ""- cluster_ca_certificate = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUV5akNDQXJLZ0F3SUJBZ0lSQU01eXFucWJNNmoxekFhbk8vczdIeVV3RFFZSktvWklodmNOQVFFTEJRQXcKRFRFTE1Ba0dBMVVFQXhNQ1kyRXdJQmNOTWpBd09USXlNakEwTWpJeFdoZ1BNakExTURBNU1qSXlNRFV5TWpGYQpNQTB4Q3pBSkJnTlZCQU1UQW1OaE1JSUNJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBZzhBTUlJQ0NnS0NBZ0VBCnc5WlYwd0hORkJiaURUcnZMQ1hJU2JoTHdScFZ1WVowMVJ5QzU0QWo1WUJjNzZyRXZIQVZlRXhWYlRYc0wrdDEKMVJPOG85bVY2UG01UXprUzdPb2JvNEV6ampTY2RBcjUvazJ4eC9BLzk0ZFl2Q20wWHZQaktXSWxiS08yaDBuNQp4VW03OFNsUks4RFVReUtGWmxTMEdXQ0hKbEhnNnpVZU8xMVB5MTQvaXBic1JYdTNFYVVLeWFJczgrekxSK2NNCm5tSU5wcHFPNUVNSzZKZVZKM0V0SWFDaFBNbHJiMVEzNHlZbWNKeVkyQmZaVkVvV2pRL1BMNFNFbGw0T0dxQjgKK3Rlc3I2TDBvOW5LT25LVlB0ZCtvSXllWnQ1QzBiMnJScnFDU1IyR09VZmsvMTV3emFiNTJ6M0JjempuV0VNOApnWWszZlBDU3JvNGE5a0xQVS9Udnk1UnZaUjJsc09mUWk3eGZpNm91dzJQeEkxc1ZPcmJnRWdza2o5Qmc4WnJYCk5GZjJpWlptSFM0djNDM1I4Q25NaHNRSVdiSmlDalhDclZCak1QbzVNS0xzNEF5U1M2dU1MelFtSjhNQWVZQlEKSHJrWEhZa21OeHlGMkhqSVlTcTdjZWFOVHJ1dTh2SFlOT2c3MGM5aGEvakZ0MXczWVl4N3NwbGRSRGpmZHZiQgpaeEtwbWNkUzY3RVNYT0dtTEhwZis1TTZXMVI3UWQwYk1SOGRQdFZJb1NmU2RZSFFLM0FDdUxrd1ZxOWpGMXlnCiswcklWMC9rN0F6bzNnUlVxeFBIV0twcHN2bFhaZCtsK0VqcTRLMnFMRXd4MlFOMDJHL1dGVUhFdGJEUXAvZWYKZGxod3Z0OHp1VklIbXE0ejlsMGZSOU9QaGN6UFpXR0dyWnUrTlQ2cm5RTUNBd0VBQWFNak1DRXdEZ1lEVlIwUApBUUgvQkFRREFnS2tNQThHQTFVZEV3RUIvd1FGTUFNQkFmOHdEUVlKS29aSWh2Y05BUUVMQlFBRGdnSUJBQjQ0CmRzbExhRzFrSzFHaFZiR1VEbWtWNndsTHJ1a1F3SHc2Vkt2azJ0T0UwMU1Hbng5SkJwVFFIL2FMdWYxUTN0WVAKaVhWSnp1bGsxOWg2MjUrcEs0ekpQcDZSWkhMV3htK2M0a1pLejlwV3F3RERuNmZqTVlKOEdMNVZFOUwvQUdSRgpscEI5ZTZNOVNGY20ra2lMVVlLazg3VG1YaGd4dzFXYXhtaDEwd01DNlZPOGNuZlVRQkJJQkVMVXhNZy9DRE9SCjZjSGFTdU5ETlg0UG80eFF3NnV4c0d1R2xDbHltOUt4Z2pIYjQ5SWp2NnN5clRIcGkrVkxmQ3d4TmdLZUwxZ1cKNURVR3ZoMmpoTVJqNDhyV3FoY3JjUHI4aWtoeVlwOXMwcFJYQVlUTk52SlFzaDhmYllOcTIzbDZwVW16dUV0UwpQS2J2WUJDVmxxa29ualZiRkxIeXJuRWNFbllqdXhwYy94bWYvT09keHhUZzlPaEtFQTRLRTQySFJ2cW1SZER5CkFldVhIcUxvUm54TXF1Z0JxL0tTclM2S0tjQW11eVJWdkhJL21MUlhmY1k1VThCWDBXcUF0N1lrWm54d1JnRkQKQndRcnEvdDJrUkMySSsxR1pUd2d1Y3hyc0VrYlVoVG5DaStVbjNDRXpTbmg5anBtdDBDcklYaDYzeC9LY014egpGM0ZXNWlnZDR4MHNxYk5oK3B4K1VSVUlsUmZlMUxDRWg3dGlraVRHRGtGT05EQXBSQUMycnUrM0I5TlpsR0hZCm9jWS9tcTlpdUtXTUpobjFHeXJzWGZLZXQrakliZzhUNzZzaEora0E4djU3VmdBdlRRSEh1YTg2SHl6d1d2Z0QKQ2ZaZFhpeURvZGlWRXhPNGlGaG00T1dhZld5U0ltSUsrOCs1Z2daZwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="- host = "https://cs-aks-f9e8be99.hcp.westeurope.azmk8s.io:443"- password = ""- username = "clusterUser_cs-rg_cs-aks"},] -> null- kube_config_raw = (sensitive value)- kubelet_identity = [- {- client_id = "fdf74b6c-68fa-4f69-b379-30375026cfee"- object_id = "4ac0dde7-d3be-4975-b7a2-9b455cb80bbd"- user_assigned_identity_id = "/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourcegroups/MC_cs-rg_cs-aks_westeurope/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cs-aks-agentpool"},] -> null- kubernetes_version = "1.17.11" -> null- location = "westeurope" -> null- name = "cs-aks" -> null- node_resource_group = "MC_cs-rg_cs-aks_westeurope" -> null- private_cluster_enabled = false -> null- private_link_enabled = false -> null- resource_group_name = "cs-rg" -> null- sku_tier = "Free" -> null- tags = {- "Environment" = "Development"} -> null- addon_profile {- aci_connector_linux {- enabled = false -> null}- azure_policy {- enabled = false -> null}- http_application_routing {- enabled = false -> null}- kube_dashboard {- enabled = false -> null}- oms_agent {- enabled = false -> null- oms_agent_identity = [] -> null}}- auto_scaler_profile {- balance_similar_node_groups = false -> null- max_graceful_termination_sec = "600" -> null- scale_down_delay_after_add = "10m" -> null- scale_down_delay_after_delete = "10s" -> null- scale_down_delay_after_failure = "3m" -> null- scale_down_unneeded = "10m" -> null- scale_down_unready = "20m" -> null- scale_down_utilization_threshold = "0.5" -> null- scan_interval = "10s" -> null}- default_node_pool {- availability_zones = [- "1",- "2",] -> null- enable_auto_scaling = true -> null- enable_node_public_ip = false -> null- max_count = 4 -> null- max_pods = 30 -> null- min_count = 2 -> null- name = "default" -> null- node_count = 2 -> null- node_labels = {} -> null- node_taints = [] -> null- orchestrator_version = "1.17.11" -> null- os_disk_size_gb = 128 -> null- tags = {} -> null- type = "VirtualMachineScaleSets" -> null- vm_size = "Standard_D2_v2" -> null- vnet_subnet_id = "/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourceGroups/cs-rg/providers/Microsoft.Network/virtualNetworks/cs-network/subnets/cs-subnet" -> null}- identity {- principal_id = "72559f39-68db-424c-b18c-b7cadb893314" -> null- tenant_id = "8f55a88a-7752-4e10-9bbb-e847ae93911d" -> null- type = "SystemAssigned" -> null}- network_profile {- dns_service_ip = "10.0.0.10" -> null- docker_bridge_cidr = "172.17.0.1/16" -> null- load_balancer_sku = "Standard" -> null- network_plugin = "azure" -> null- network_policy = "calico" -> null- outbound_type = "loadBalancer" -> null- service_cidr = "10.0.0.0/16" -> null- load_balancer_profile {- effective_outbound_ips = [- "/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourceGroups/MC_cs-rg_cs-aks_westeurope/providers/Microsoft.Network/publicIPAddresses/490fd61a-dc70-4104-bed3-533a69c723f3",] -> null- idle_timeout_in_minutes = 0 -> null- managed_outbound_ip_count = 1 -> null- outbound_ip_address_ids = [] -> null- outbound_ip_prefix_ids = [] -> null- outbound_ports_allocated = 0 -> null}}- role_based_access_control {- enabled = true -> null- azure_active_directory {- admin_group_object_ids = [] -> null- client_app_id = "f9bf8772-aaba-4773-a815-784b31f9ab8b" -> null- managed = false -> null- server_app_id = "fa7775b3-ea31-4e99-92f5-8ed0bac3e6a8" -> null- server_app_secret = (sensitive value)- tenant_id = "8f55a88a-7752-4e10-9bbb-e847ae93911d" -> null}}- windows_profile {- admin_username = "azureuser" -> null}}# azurerm_resource_group.demo will be destroyed- resource "azurerm_resource_group" "demo" {- id = "/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourceGroups/cs-rg" -> null- location = "westeurope" -> null- name = "cs-rg" -> null- tags = {} -> null}# azurerm_subnet.demo will be destroyed- resource "azurerm_subnet" "demo" {- address_prefix = "10.1.0.0/22" -> null- address_prefixes = [- "10.1.0.0/22",] -> null- enforce_private_link_endpoint_network_policies = false -> null- enforce_private_link_service_network_policies = false -> null- id = "/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourceGroups/cs-rg/providers/Microsoft.Network/virtualNetworks/cs-network/subnets/cs-subnet" -> null- name = "cs-subnet" -> null- resource_group_name = "cs-rg" -> null- service_endpoints = [] -> null- virtual_network_name = "cs-network" -> null}# azurerm_virtual_network.demo will be destroyed- resource "azurerm_virtual_network" "demo" {- address_space = [- "10.1.0.0/16",] -> null- dns_servers = [] -> null- guid = "89184266-b65e-423f-a0af-12ed8c2131f2" -> null- id = "/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourceGroups/cs-rg/providers/Microsoft.Network/virtualNetworks/cs-network" -> null- location = "westeurope" -> null- name = "cs-network" -> null- resource_group_name = "cs-rg" -> null- subnet = [- {- address_prefix = "10.1.0.0/22"- id = "/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourceGroups/cs-rg/providers/Microsoft.Network/virtualNetworks/cs-network/subnets/cs-subnet"- name = "cs-subnet"- security_group = ""},] -> null- tags = {} -> null}Plan: 0 to add, 0 to change, 4 to destroy.Changes to Outputs:- kube_config = <<~EOTapiVersion: v1clusters:- cluster:certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUV5akNDQXJLZ0F3SUJBZ0lSQU01eXFucWJNNmoxekFhbk8vczdIeVV3RFFZSktvWklodmNOQVFFTEJRQXcKRFRFTE1Ba0dBMVVFQXhNQ1kyRXdJQmNOTWpBd09USXlNakEwTWpJeFdoZ1BNakExTURBNU1qSXlNRFV5TWpGYQpNQTB4Q3pBSkJnTlZCQU1UQW1OaE1JSUNJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBZzhBTUlJQ0NnS0NBZ0VBCnc5WlYwd0hORkJiaURUcnZMQ1hJU2JoTHdScFZ1WVowMVJ5QzU0QWo1WUJjNzZyRXZIQVZlRXhWYlRYc0wrdDEKMVJPOG85bVY2UG01UXprUzdPb2JvNEV6ampTY2RBcjUvazJ4eC9BLzk0ZFl2Q20wWHZQaktXSWxiS08yaDBuNQp4VW03OFNsUks4RFVReUtGWmxTMEdXQ0hKbEhnNnpVZU8xMVB5MTQvaXBic1JYdTNFYVVLeWFJczgrekxSK2NNCm5tSU5wcHFPNUVNSzZKZVZKM0V0SWFDaFBNbHJiMVEzNHlZbWNKeVkyQmZaVkVvV2pRL1BMNFNFbGw0T0dxQjgKK3Rlc3I2TDBvOW5LT25LVlB0ZCtvSXllWnQ1QzBiMnJScnFDU1IyR09VZmsvMTV3emFiNTJ6M0JjempuV0VNOApnWWszZlBDU3JvNGE5a0xQVS9Udnk1UnZaUjJsc09mUWk3eGZpNm91dzJQeEkxc1ZPcmJnRWdza2o5Qmc4WnJYCk5GZjJpWlptSFM0djNDM1I4Q25NaHNRSVdiSmlDalhDclZCak1QbzVNS0xzNEF5U1M2dU1MelFtSjhNQWVZQlEKSHJrWEhZa21OeHlGMkhqSVlTcTdjZWFOVHJ1dTh2SFlOT2c3MGM5aGEvakZ0MXczWVl4N3NwbGRSRGpmZHZiQgpaeEtwbWNkUzY3RVNYT0dtTEhwZis1TTZXMVI3UWQwYk1SOGRQdFZJb1NmU2RZSFFLM0FDdUxrd1ZxOWpGMXlnCiswcklWMC9rN0F6bzNnUlVxeFBIV0twcHN2bFhaZCtsK0VqcTRLMnFMRXd4MlFOMDJHL1dGVUhFdGJEUXAvZWYKZGxod3Z0OHp1VklIbXE0ejlsMGZSOU9QaGN6UFpXR0dyWnUrTlQ2cm5RTUNBd0VBQWFNak1DRXdEZ1lEVlIwUApBUUgvQkFRREFnS2tNQThHQTFVZEV3RUIvd1FGTUFNQkFmOHdEUVlKS29aSWh2Y05BUUVMQlFBRGdnSUJBQjQ0CmRzbExhRzFrSzFHaFZiR1VEbWtWNndsTHJ1a1F3SHc2Vkt2azJ0T0UwMU1Hbng5SkJwVFFIL2FMdWYxUTN0WVAKaVhWSnp1bGsxOWg2MjUrcEs0ekpQcDZSWkhMV3htK2M0a1pLejlwV3F3RERuNmZqTVlKOEdMNVZFOUwvQUdSRgpscEI5ZTZNOVNGY20ra2lMVVlLazg3VG1YaGd4dzFXYXhtaDEwd01DNlZPOGNuZlVRQkJJQkVMVXhNZy9DRE9SCjZjSGFTdU5ETlg0UG80eFF3NnV4c0d1R2xDbHltOUt4Z2pIYjQ5SWp2NnN5clRIcGkrVkxmQ3d4TmdLZUwxZ1cKNURVR3ZoMmpoTVJqNDhyV3FoY3JjUHI4aWtoeVlwOXMwcFJYQVlUTk52SlFzaDhmYllOcTIzbDZwVW16dUV0UwpQS2J2WUJDVmxxa29ualZiRkxIeXJuRWNFbllqdXhwYy94bWYvT09keHhUZzlPaEtFQTRLRTQySFJ2cW1SZER5CkFldVhIcUxvUm54TXF1Z0JxL0tTclM2S0tjQW11eVJWdkhJL21MUlhmY1k1VThCWDBXcUF0N1lrWm54d1JnRkQKQndRcnEvdDJrUkMySSsxR1pUd2d1Y3hyc0VrYlVoVG5DaStVbjNDRXpTbmg5anBtdDBDcklYaDYzeC9LY014egpGM0ZXNWlnZDR4MHNxYk5oK3B4K1VSVUlsUmZlMUxDRWg3dGlraVRHRGtGT05EQXBSQUMycnUrM0I5TlpsR0hZCm9jWS9tcTlpdUtXTUpobjFHeXJzWGZLZXQrakliZzhUNzZzaEora0E4djU3VmdBdlRRSEh1YTg2SHl6d1d2Z0QKQ2ZaZFhpeURvZGlWRXhPNGlGaG00T1dhZld5U0ltSUsrOCs1Z2daZwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==server: https://cs-aks-f9e8be99.hcp.westeurope.azmk8s.io:443name: cs-akscontexts:- context:cluster: cs-aksuser: clusterUser_cs-rg_cs-aksname: cs-akscurrent-context: cs-akskind: Configpreferences: {}users:- name: clusterUser_cs-rg_cs-aksuser:auth-provider:config:apiserver-id: fa7775b3-ea31-4e99-92f5-8ed0bac3e6a8client-id: f9bf8772-aaba-4773-a815-784b31f9ab8bconfig-mode: "1"environment: AzurePublicCloudtenant-id: 8f55a88a-7752-4e10-9bbb-e847ae93911dname: azureEOT -> nullDo you really want to destroy all resources?Terraform will destroy all your managed infrastructure, as shown above.There is no undo. Only 'yes' will be accepted to confirm.Enter a value: yesazurerm_kubernetes_cluster.demo: Destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourcegroups/cs-rg/providers/Microsoft.ContainerService/managedClusters/cs-aks]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 10s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 20s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 30s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 40s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 50s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 1m0s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 1m10s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 1m20s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 1m30s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 1m40s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 1m50s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 2m0s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 2m10s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 2m20s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 2m30s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 2m40s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 2m50s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 3m0s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 3m10s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 3m20s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 3m30s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 3m40s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 3m50s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 4m0s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 4m10s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 4m20s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 4m30s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 4m40s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 4m50s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 5m0s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 5m10s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 5m20s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 5m30s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 5m40s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 5m50s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 6m0s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 6m10s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 6m20s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 6m30s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 6m40s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 6m50s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 7m0s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 7m10s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 7m20s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 7m30s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 7m40s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 7m50s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 8m0s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 8m10s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 8m20s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 8m30s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 8m40s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 8m50s elapsed]azurerm_kubernetes_cluster.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...ontainerService/managedClusters/cs-aks, 9m0s elapsed]azurerm_kubernetes_cluster.demo: Destruction complete after 9m5sazurerm_subnet.demo: Destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourceGroups/cs-rg/providers/Microsoft.Network/virtualNetworks/cs-network/subnets/cs-subnet]azurerm_subnet.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...lNetworks/cs-network/subnets/cs-subnet, 10s elapsed]azurerm_subnet.demo: Destruction complete after 11sazurerm_virtual_network.demo: Destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourceGroups/cs-rg/providers/Microsoft.Network/virtualNetworks/cs-network]azurerm_virtual_network.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-...oft.Network/virtualNetworks/cs-network, 10s elapsed]azurerm_virtual_network.demo: Destruction complete after 11sazurerm_resource_group.demo: Destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourceGroups/cs-rg]azurerm_resource_group.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourceGroups/cs-rg, 10s elapsed]azurerm_resource_group.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourceGroups/cs-rg, 20s elapsed]azurerm_resource_group.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourceGroups/cs-rg, 30s elapsed]azurerm_resource_group.demo: Still destroying... [id=/subscriptions/a7a456e9-0307-4196-b786-5a33ce52b5fd/resourceGroups/cs-rg, 40s elapsed]azurerm_resource_group.demo: Destruction complete after 48sDestroy complete! Resources: 4 destroyed.
Summary
Availability zones, Azure AD integration, and Calico network policies all help to achieve high availability, seamless identity management, and advanced network traffic management for applications deployed in AKS.
Availability zones help protect your workloads from Azure data center failures and ensure production system resiliency. Azure AD integration is crucial for unifying the identity management of the cluster, as customers can continue to leverage their investments in Azure AD for managing AKS workloads as well. Calico network policy helps enhance security posture of line-of-business applications deployed in AKS by ensuring that only legit traffic reaches your workloads.
These features are key for ensuring the production readiness of your AKS cluster.
As a next step, the automated deployment of the AKS cluster covered in this article can also be integrated with your existing infrastructure-as-code DevOps pipelines for production-scale deployments.
