Cleanup in Bash Scripts

[Brain dump so I don’t forget this one]

So you want your bash script to exit on an error but you’d like it to clean some stuff up before it closes after the error occurs.

No problem a TRAP can do this for you (read detailed docs for caveats).

In a very simple form it looks like this:

set -e
function cleanup()
echo -e "—-> Trap caught! Do cleanup here"
trap cleanup EXIT
# imagine some stuff happens here
# and it exists
exit 1
view raw trap.sh hosted with ❤ by GitHub
Using Trap to fire cleanup on exit

Learn more here: https://www.linuxjournal.com/content/bash-trap-command


Azure Databricks and Terraform: Create a Cluster and PAT Token

[Update Feb 2021] There is now a Terraform Provider for Databricks, it’s a better route – https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs

My starting point for a recent bit of work was to try and reliably and simply deploy and manage Databricks clusters in Azure. Terraform was already in use so I set about trying to see how I could use that to also manage Databricks.

I had a look around and after trying the Terraform REST provider and a third party Datbricks provider (didn’t have much luck with either) found a Terraform Shell provider. This turned out to be exactly what I needed.

If you haven’t written a Terraform provider here’s a crash course. You basically just define a method for create, read, update and delete and the parameters they take. Then Terraform does the rest.

The Shell provider (https://github.com/scottwinkler/terraform-provider-shell) lets you do this by passing in scripts (bash, powershell, any executable that can take stdin and output stdout). In this case I wrote some powershell to wrap the databricks-cli.

It’s better (or different) to localexec with nullresources as you can store information in the Terraform State and detect drift. If a read returns different information than the current information in the state then update will be called, for example.

So I took the work of Alexandre and wrapped it into this provider and using the Shell provider have a simple, no frills Databricks provider for Terraform which makes calls to Databricks via the databricks-cli.

This is currently a simple hack and hasn’t undergone any significant testing: https://github.com/lawrencegripper/hack-databricksterraform. The flow is as follows:

Hopefully this might be useful to others as a starting point for others.


Generate docker images of specific size

For some testing I’m doing I need a set of images of a specific size to simulate pulling larger vs smaller image.

Here is a quick script I put together for generating a 200mb, 600mb, 1000mb and 2000mb image (tiny bit larger as alpine included). Took a while to work out best to use /dev/urandom not /dev/zero as with zero the images got compressed for transfer.

set -e
set -x
# Push 200mb image
dd if=/dev/urandom of=./file.bin bs=1M count=200
docker build -t lawrencegripper/big:200mb .
docker push lawrencegripper/big:200mb
rm ./file.bin
# Push 600mb image
dd if=/dev/urandom of=./file.bin bs=1M count=600
docker build -t lawrencegripper/big:600mb .
docker push lawrencegripper/big:600mb
rm ./file.bin
# Push 1000mb image
dd if=/dev/urandom of=./file.bin bs=1M count=1000
docker build -t lawrencegripper/big:1000mb .
docker push lawrencegripper/big:1000mb
rm ./file.bin
# Push 2000mb image
dd if=/dev/urandom of=./file.bin bs=1M count=2000
docker build -t lawrencegripper/big:2000mb .
docker push lawrencegripper/big:2000mb
rm ./file.bin
view raw 1_gen_image_sizes.sh hosted with ❤ by GitHub
FROM alpine
COPY ./file.bin .
view raw 2_Dockerfile hosted with ❤ by GitHub