Azure, How to

Using .img files on Azure Files to get full Linux filesystem compatibility

Here is the scenario, I wanted to use a container to run a linux app in Azure and I need to persist changes to the filesystem via Azure Files. This, initially, appears to be nice and simple – you mount the Azure File share using CIFS and then pass this into docker as a volume mount. But what if you app relies on some linux specific operations some, like symlinks, can be emulated see here for details, but others can’t.

This is a really simple little hack which I learnt from the Azure Cloudshell implementation. It uses the ‘.img’ format to store a full EXT2 filesystem on Azure files.

How does it work? Well if you open a cloudshell instance and use the ‘mount’ command you can see it has two mounts, one for CIFS and one for a loop0.

Seeing this started to peak my interest, what was the loop0 device mounting as my home directory? The cloudshell creates a storage account to persist your files between sessions, next I took a look at this to see what I could find.

imgmount2

This is when I found the ‘.img’ file being used. So how do we use this approach for ourselves, as it seems to work nicely for cloudshell?

It’s actually pretty simple, we mount the Azure file share with CIFS, create the ‘.img’ file in the CIFS share, format it and then mount the ‘.img’ file. Done.

The key is to create a ‘img’ file which is sparse, meaning we don’t write all the empty space to file storage, otherwise creating a 10Gig ‘img’ file involves copying 10gig to Azure files. This is done by passing in the ‘seek’ command into dd on like 15.

So this gives you a fully compatible Linux disk stored on Azure files so it can persist container restarts and machine moves.

Ps. If you’re replicating super critical data dive this some extensive testing first, this approach worked nicely for my use case but do exercise caution.

Standard
How to

Quick How to: Mount Azure Files Shares with Symlinks support on Ubuntu

By default mounting Azure File Shares on linux using CIFS doesn’t enable support for symlinks. You’ll see an error link this:

auser@acomputer:/media/shared$ ln -s linked -n t
ln: failed to create symbolic link 't': Operation not supported

So how do you fix this, simple? Simple add the following to the end of your CIFS mount command:

,mfsymlinks

So the command will look something like:

sudo mount -t cifs //<your-storage-acc-name>.file.core.windows.net/<storage-file-share-name> /media/shared -o vers=3.0,username=<storage-account-name>,password='<storage-account-key>',dir_mode=0777,file_mode=0777,mfsymlinks

So what does this do? Well you have to thank Steve French and Conrad Minshal. They defined a format for storing symlinks on SMB shares, an explanation of the format can be found here.

Thanks to renash for her comment (scroll to the bottom) which enabled me to find this, blog is to help others and give more details.

Standard
kubernetes

Kubernetes: Prevent Container from accessing Cluster API

I’ve recently been playing with Kubernetes as way to efficiently host my microservices. It’s great and I’m really enjoying it.

One thing that did take me by surprise is that Containers, by default, have credentials mounted which allow them to talk to the cluster API. This is incredibly useful for pulling information about other services, secrets or enabling self-orchestrating/scaling services, however, for the services that don’t need this it, it presents on opportunity for an attacker to escalate their privileges.

We tend to think about the containers (roughly) providing limited access to resources, information and preventing access to other containers but with this token the container is able to reach beyond the container. The mitigation is that, first, the attacker would have to compromise the application running in the container, once they had done this they could extract and use the token to control or pull information from your cluster.

As always, defense in depth is a good policy. We aim to prevent our container from being compromised but we ALSO aim to limit the damage if it is.

So here is how to prevent that service account being mounted into your containers which don’t require it:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  serviceAccountName: build-robot
  automountServiceAccountToken: false
  ...

RBAC also presents an opportunity to have finer grained control over the level of access and is almost certainly something to consider too along with the use of namespaces.

How could an attacker get access?

Here is a really simple example of a badly written nodejs app which allows the users to pass in a version, this is used to find a file and the contents of the file is then returned to the user.

The k8s token is mounted at /var/run/secrets/kubernetes.io/serviceaccount

Using the ‘../’ syntax we can traverse out of the apps folder and down to the root then we can request the token, ca and namespace files from the service account.

This is simple example to show the files exist and can be leaked. Other vulnerabilities with remote code execution would allow the attacker to make requests to the cluster API using this token.

Summary

Kubernetes has a great architecture which lets containers talk to the cluster API but in some scenarios, when this access isn’t required, you’re likely to want to turn this off to provide defense in depth by the principal of least privileged.

 

Standard