AKS Network Policies

Earlier this week my girlfriend (who is a networking ninja) asked a great question, how do you secure traffic between pods in Azure Kubenetes Service (AKS).

This simple sounding question started a two-day hack to allow us to experiment with deploying AKS and configuring network policies. As it turns out, this functionality is currently in preview, so it wasn’t as simple as it might have first appeared.

In this post, I’m going to outline the basics of a demo that demonstrates the process. If you’re interested in following this tutorial then you’ll be pleased to hear you don’t have to install any tools locally!

Cloud Shell

The Cloud Shell is a browser-based shell experience available anywhere, providing you can log into the Azure portal. With the Cloud Shell, you are able to pick between either Bash or Powershell, which is great for me as a long time macOS user now sporting a Dell.

Cloud Shell Scripts

When I first saw the Cloud Shell, I assumed it was only good for executing individual commands but its just as powerful as PowerShell on my Dell or Bash on my Mac. This is because it’s backed by Azure File Storage, which allows us to upload and execute existing shell scripts.

Using the Cloud Shell with scripts makes it super easy to quickly deploy services in a reproducible way, without having to install and configure any tools on your local development machine, which is a HUGE win!

Preview Features

This feature is in preview, so before we start, we need to enable it. In the Cloud Shell, enter the following:

az feature register --name EnableNetworkPolicy --namespace Microsoft.ContainerService

This can take a few minutes but only ever needs to be done just the once! Once It’s completed, you can query its status using the following:

az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/EnableNetworkPolicy')].{Name:name,State:properties.state}

Once registered is success, you’ll need to run the following to finish up:

az provider register -n Microsoft.ContainerService

Creating an AKS Instance

Now for the interesting bits! We’re going to deploy a new AKS cluster! Because we’ll be using the same names throughout the process, we’ll go ahead and define these as variables.

export RESOURCE_GROUP_NAME=my-aks-demo
export CLUSTER_NAME=my-AKS1

Create a resource group

az group create --name $RESOURCE_GROUP_NAME --location eastus

Create a Virtual Network & Subnet

az network vnet create --resource-group $RESOURCE_GROUP_NAME --name myVnet --address-prefixes 10.0.0.0/8 --subnet-name myAKSSubnet --subnet-prefix 10.240.0.0/16

We get the Virtual Network Subnet Resource ID

SUBNET_ID=$(az network vnet subnet show --resource-group $RESOURCE_GROUP_NAME --vnet-name myVnet --name myAKSSubnet --query id -o tsv) 

az aks create --resource-group $RESOURCE_GROUP_NAME --name $CLUSTER_NAME --kubernetes-version 1.12.4 --network-plugin azure --service-cidr 10.0.0.0/16 --dns-service-ip 10.0.0.10 --docker-bridge-address 172.17.0.1/16 --vnet-subnet-id $SUBNET_ID --network-policy calico --generate-ssh-keys

Nows the time to pop your feet up and relax as this is going to take a while. On average it takes about 10 minutes to finish. Once it has though, it’ll have spun up a new AKS cluser which means we can grab a copy of the Kubernetes Context using Kubectl:

az aks get-credentials --resource-group $RESOURCE_GROUP_NAME --name $CLUSTER_NAME

Deploying something!

We need to deploy something to AKS in order to confirm that the network policies are working. We don’t need anything too complex so a NGINX Container it is! We’ll give it a label of “app=api” and expose the pod.

kubectl run api --image=nginx --labels app=api --expose --port 80

By default, any other containers should be able to communicate with the API we just deployed, so lets confirm! To do that,we’ll go ahead and deploy another container.
kubectl run --rm -it --image=alpine test-np

Testing, Testing, 123

200.gif

Lets test if it works! To do this, we can use the prompt and enter the following:
wget http://api -qO-

We should then get some HTML passed back as text.

Deny Policy

gandalfmeme

So we know we can communicate, which is default behavior, lets deploy a network policy which will deny the communication. The policies are super simple, in fact the sample here is only 9 LOC!

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: api-policy
spec:
podSelector:
matchLabels:
app: api
ingress: []

With this saved to disk (or uploaded to the Cloud Shell), we can call the following:
kubectl create -f [RootDirectory]/deny.yaml

Testing

To retest we’re going to need to reattach to the pod. To do this, enter the following:
kubectl attach [test-np name] -i -t

We can then repeat the process above but adding a timeout to ensure we don’t wait until boredom or death occurs before being told it failed. (its default is 30 seconds…)

wget -qO- --timeout=2 http://api

Wrap Up

You’re now a cutting edge, AKS networking ninja!

Well, perhaps not exactly but you’ve at least learn the basic of this new feature to AKS. To learn more about this new feature then check out the offical documentation on the Microsoft site.

Huge thanks and credit goes to Simona Tarantola & Justin Davies. This post wouldn’t have been possible without them!

App Services Custom Domain, SSL & DNS

We’ve all seen tutorials which demonstrate how to deploy a simple todo list backend to Azure but how many have you read that go onto secure it? In this post, I’m going to cover how I’m securing the Bait News v2 backend infrastructure as well as covering how to configure custom domains.

Why bother?

Apple announced in 2015 that Apps and their corresponding backend servers would need to support App Transport Security (ATS).

ATS was introduced with iOS 9 as a security enhancement to ensure all connections by apps use HTTPs. Initially slated to go into effect for all new app store submissions from January 2017, it has since been postponed with no update on when it’ll be coming into effect. Although the requirement has been delayed, it’s still something that all app developers should be implementing as it provides our users with added security, making man in the middle attacks impossible to go unnoticed.

Historically, you’ll see most developers (including myself), opt to turn ATS off to make our lives easier. Some will take a lighter touch and only disable ATS for a single domain (they’re backend) which is not much more secure than turning ATS off altogether. Either approach opens up your users and data to attack and should be avoided.

So what do we need to do to secure our app? Lets first register a domain for our backend.

Custom Domains

DNS

I’ve been using 123-Reg as my domain registrar for 10 years and continue to use them as I migrate my websites to Azure. Most domain registrars will also provide some basic DNS functionality but you would normally want to use a 3rd party DNS Service for more advance situations. In my case, I’m using 123-Regs DNS service and have added a number of CNAMEs pointing to Azure.

Adding records

Below you see the minimum required records needed to enable my custom domain.

Permanent Records

Screen Shot 2017-09-02 at 15.20.53

Temporary Records

Screen Shot 2017-09-02 at 15.32.15

To get started, I have added an A record pointing to the App Service instance using its IP address. You can find your App Service IP address by going into it’s Custom Domain blade within the Azure portal.

Once you’ve added the A record, you can then create the CNAME which will map www requests to your backends url. You can find your destination in the Overview blade of the App Service.

Verify Domain Ownership

Azure needs to know I own the domain I’m trying to map. To prove this, I’ll add two records to my DNS settings which are the temporary records listed above.

Once I’ve added the verify CNAME records, I can save and sit tight. DNS Records need to propagate across the globe, which can take up to 24 hours.

This is end result of what my DNS setting configuration looked like. I also created some CNAMEs to redirect traffic from subdomains to other Azure services.

Screen Shot 2017-08-17 at 11.50.36

Portal Configuration

To finish off, I need to configure the App Service Custom Domain settings.

Screen Shot 2017-09-02 at 15.53.03.png

Hit the ‘Add Hostname’ button and enter the custom domain.

Screen Shot 2017-09-02 at 15.54.03

After hitting Validate, Azure will check the DNS records to confirm the domain exists and that you own it. You should see something like this.

Screen Shot 2017-09-02 at 15.55.53

Hitting ‘Add hostname’ will complete the process of configuring a custom domain for your App Service. If you’re deploying a mobile backend, you may want to create CNAME record which maps api.domain.net to your mobile backend and whilst keeping www.domain.net mapped to a ASP.NET website.

Adding Security

SSL Certificates

As mentioned at the start of this post, enabling HTTPS prevents MITM attacks and ensures your communication between server and client is secure. Its pretty straight forward to enable within App Services but much like DNS, it can take a while (but this time its human factors rather than waiting for computers to sync up).

First things first, You’ll need to purchase a certificate.  I opted to 123-Reg as they provide a few options to meet most users requirements and its integration with my domain management make it a no brainer to use.

I should admit that I did make a mistake when I first purchase a certificate, which caused a few days of delays, so its important to double check the type of certificate you’re purchasing. I had purchased a certificate which was for only www.baitnews.io. This mean that my mobile api of api.baitnews.io couldn’t use the certificate. 123-Reg refunded the first certificate and I tried again, but this time making sure to purchase a certificate which support unlimited subdomains. You can see below the original certificate has been revoked and the new certificate supports wildcards.

When you apply for a certificate, you’ll be provided a download which includes your certificate request (CSR) in a PEM format. You also get the private key which you’ll use later to create a new certificate.

Screen Shot 2017-09-02 at 16.13.32

Once you’ve been issued the certificate, you’re ready to create a new certificate which you’ll use in Azure for everything. This is a pretty easy process as we can use OpenSSL on almost any platform. I’m on a Mac but this works the same on both Windows and Linux.

openssl pkcs12 -export -out baitnews.pfx -inkey
/Users/michaeljames/Downloads/SSL-CSR5/private-key.key -in
/Users/michaeljames/Desktop/wildcard.cert

cert

Variables

  • [Output file name] -| What do you want to call the certificate?
  • [private-key.key path] The location of the private key. This would have been provided when requesting the certificate.
  • [wildcard.cert path] The location of the freshly issued certificate.

Once you press enter, you’ll need to type in a couple of passwords and then you’ll be set. It’ll look something like this:

Screen Shot 2017-09-02 at 16.38.43

You now have your certificate ready for uploading to Azure. The conversion of certificates isn’t the easiest of procedures to wrap your head around on the first few goes. If you’re worried about this step then keep in mind you can purchase SSL certificates through the Azure Portal, which skips many of the above steps! It does however add a small premium to the cost involved in securing your backend as you’ll find the certificate a little more expensive but your also required to store it in KeyVault.

Binding Domains with Certificates

Lets upload our new certificate to our App Service. To do this, head over to SSL Certificate blade and hit ‘Upload Certificate’. You’ll need to provide the password used to create the certificate.

Screen Shot 2017-09-07 at 12.26.17.png

If successful, you’ll see your certificate as been imported and is ready to use with your custom domains.

Screen Shot 2017-09-07 at 12.29.33.png

Add Binding

The last step is to bind out SSL certificate with our custom domain. Clicking ‘Add Binding’ will allow you to select both the custom domain and SSL from a drop down.

Screen Shot 2017-09-07 at 12.29.40

Hitting Add Binding will finish the process. You now have a custom domain mapped your App Service instance that supports HTTPS. Any users visiting your backend will be greeted  with the familiar green padlock in the address bar.

Screen Shot 2017-09-07 at 12.31.59.png

Wrapping Up

Adding custom domains and enabling secure connectivity between your mobile app and backend is extremely simple and theres no good reason not to enable it (unless you’re hacking on a demo or POC).

In the next post I’m going to cover how to expand our setup to to route traffic to the nearest App Service instance.