Azure, Coding

From ARM to Terraform: Why I’m becoming a Terraform convert

Let me give you some background: I’ve been a heavy user of the Azure Resource Manager (ARM) templates for some time now. We’ve had some good times together. We’ve also had some bad times and boy have we seriously fallen out at times!

I’ve used ARM extensively to automate deployment, configuration and update of the infrastructure in Azure. I’ve used it on everything from single-person projects to large scale dual-region deployments of services with critical live traffic.

The takeaway for me has been: ARM is solid, reliable and does what I need – however, it isn’t always the easiest to use and it forces a wall between infrastructure deployment and application configuration which I’ve always found to be painful.

For example, when working with Azure Storage through ARM you can create a storage account, but you can’t create ‘queues’ or upload ‘blobs’ into the account. Another example – you can create an Azure Kubernetes cluster with AKS but you can’t create ‘secrets’ or ‘deployments’ in the cluster.

Shortcomings like these can be painful at times, and difficult to work around.

Typically, an application running in AKS might use a managed Redis Cache instance or an Azure Storage queue. Ideally, you want to use ARM to…

  1. Create your Redis cache
  2. Create your storage queue
  3. Create your AKS cluster
  4. Create secretes in AKS cluster with Redis and Storage access keys
  5. Deploy your container based application.

Once ARM has finished, you want your application set and and ready to go. You don’t want the container to have to check and create queues if they don’t exist. You don’t want it to somehow retrieve the Redis access keys.

But, because you can’t set these things up with ARM you’re unable to deploy a working application in one hit. You have to run ARM to create Redis, AKS and storage then after that you have to run a script which injects the access keys into your AKS cluster so your app can use them. That’s not ideal. (Some interesting integration work between ARM and Terraform is underway which provides an interesting way to merge both but I’m not going into that now. Read more here.)

My other issue with ARM is much more personal and many may disagree, but I find it very hard to author. VSCode has introduced some great ARM tooling which certainly helps make things better, I still find it dense and hard to parse mentally.

ARMvsHCL

ARM (left) vs Terraform (right): VM Creation in Azure

So, what’s the solution?

Recently, I’ve been working with Terraform and I think it helps with a lot of these issues. First up, in my opinion, it’s nice and simple to read. Here is an example creating a storage account and uploading a file into it:

storageAndUpload

Create a storage account with a random name, create a container called ‘scripts’ in it and upload ‘init.sh’ script into the container. Full example here generates a SAS url to the uploaded script.

Terraform is run on your client, which means you can also do more; things that aren’t possible with ARM, such as uploading ‘blobs’ or configuring ‘queues’.

But that’s not really the main thing that has me converted. Terraform is made up of ‘Providers’ which each know how to talk to a set of API’s. Think of these a bit like ‘Resource Providers’ in ARM. Terraform has an Azure provider but it also has a Kuberenetes provider, a CloudFlare provider, a RabbitMQ provider… you get the point.

This is where things really light up. No modern solution of any serious scale is solely confined to one stack/cloud… with Terraform you can reach into all of these by including, or even writing your own, providers. The result is that you have one toolchain and one interface to work with. You can bridge between infrastructure and runtime, taking access keys and injecting them into your k8s cluster as a secret, for example.

You can see an end-to-end example of this with an AKS cluster in my repo here.

“Well that all sounds great Lawrence but what’s the catch?”

Good question. There are, as they’re always are, some downsides to Terraform. Firstly, not all of ARM APIs are provided. There is a nice workaround to this which lets you embed ARM within Terraform which goes a long way to mitigate this issue. You can see an example of that here where I call Azure Batch from Terraform. Some of the docs and examples of Azure are out-of-date or need some tweaking too.  But, it’s an OSS effort so you can (and I have) add functionality for resources that are missing and fix up docs as you go.

Secondly, like any abstraction there are leaks occasionally and where you gain those abstractions it will, ultimately, bite or nip or maybe it won’t, but this depends on what you’re doing and how you’re doing it. For me this hasn’t been a problem…yet.

What else is in this space?

https://pulumi.io/ also just launched which has an interesting take on the deployment/configuration problem but I haven’t yet had a chance to play with it in detail.

Big thanks to @martinpeck for helping with this post!

Standard
Azure, How to

Using .img files on Azure Files to get full Linux filesystem compatibility

Here is the scenario, I wanted to use a container to run a linux app in Azure and I need to persist changes to the filesystem via Azure Files. This, initially, appears to be nice and simple – you mount the Azure File share using CIFS and then pass this into docker as a volume mount. But what if you app relies on some linux specific operations some, like symlinks, can be emulated see here for details, but others can’t.

This is a really simple little hack which I learnt from the Azure Cloudshell implementation. It uses the ‘.img’ format to store a full EXT2 filesystem on Azure files.

How does it work? Well if you open a cloudshell instance and use the ‘mount’ command you can see it has two mounts, one for CIFS and one for a loop0.

Seeing this started to peak my interest, what was the loop0 device mounting as my home directory? The cloudshell creates a storage account to persist your files between sessions, next I took a look at this to see what I could find.

imgmount2

This is when I found the ‘.img’ file being used. So how do we use this approach for ourselves, as it seems to work nicely for cloudshell?

It’s actually pretty simple, we mount the Azure file share with CIFS, create the ‘.img’ file in the CIFS share, format it and then mount the ‘.img’ file. Done.

The key is to create a ‘img’ file which is sparse, meaning we don’t write all the empty space to file storage, otherwise creating a 10Gig ‘img’ file involves copying 10gig to Azure files. This is done by passing in the ‘seek’ command into dd on like 15.

So this gives you a fully compatible Linux disk stored on Azure files so it can persist container restarts and machine moves.

Ps. If you’re replicating super critical data dive this some extensive testing first, this approach worked nicely for my use case but do exercise caution.

Standard
How to

Quick How to: Mount Azure Files Shares with Symlinks support on Ubuntu

By default mounting Azure File Shares on linux using CIFS doesn’t enable support for symlinks. You’ll see an error link this:

auser@acomputer:/media/shared$ ln -s linked -n t
ln: failed to create symbolic link 't': Operation not supported

So how do you fix this, simple? Simple add the following to the end of your CIFS mount command:

,mfsymlinks

So the command will look something like:

sudo mount -t cifs //<your-storage-acc-name>.file.core.windows.net/<storage-file-share-name> /media/shared -o vers=3.0,username=<storage-account-name>,password='<storage-account-key>',dir_mode=0777,file_mode=0777,mfsymlinks

So what does this do? Well you have to thank Steve French and Conrad Minshal. They defined a format for storing symlinks on SMB shares, an explanation of the format can be found here.

Thanks to renash for her comment (scroll to the bottom) which enabled me to find this, blog is to help others and give more details.

Standard