#terraform

Azure Databricks and Terraform: Create a Cluster and PAT Token

My starting point for a recent bit of work was to try and reliably and simply deploy and manage Databricks clusters in Azure. Terraform was already in use so I set about trying to see how I could use that to also manage Databricks.

I had a look around and after trying the Terraform REST provider and a third party Datbricks provider (didn’t have much luck with either) found a Terraform Shell provider. This turned out to be exactly what I needed.

If you haven’t written a Terraform provider here’s a crash course. You basically just define a method for create, read, update and delete and the parameters they take. Then Terraform does the rest.

The Shell provider (https://github.com/scottwinkler/terraform-provider-shell) lets you do this by passing in scripts (bash, powershell, any executable that can take stdin and output stdout). In this case I wrote some powershell to wrap the databricks-cli.

It’s better (or different) to localexec with nullresources as you can store information in the Terraform State and detect drift. If a read returns different information than the current information in the state then update will be called, for example.

So I took the work of Alexandre and wrapped it into this provider and using the Shell provider have a simple, no frills Databricks provider for Terraform which makes calls to Databricks via the databricks-cli.

This is currently a simple hack and hasn’t undergone any significant testing: https://github.com/lawrencegripper/hack-databricksterraform. The flow is as follows:

Hopefully this might be useful to others as a starting point for others.

Standard
How to

Writing Bash doesn’t have to be as painful as you think! Shellcheck to the rescue.

So I’ve found myself writing lots of bash scripts recently and because they tend to do real things to the file system or cloud services they’re hard to test… it’s painful.

neverfear

So it turns out there is an awesome linter/checker for bash called shellcheck which you can use to catch a lot of those gotchas before they become a problem.

There is a great plugin for vscode so you get instant feedback when you do something you shouldn’t.

Better still it’s easy to get running in your build pipeline to keep everyone honest. Here is an example task for Azure Devops to run it on all scripts in the ./scripts folder.

Next on my list is to play with the xunit inspired testing framework for bash called shunit2 but kinda feel if you have enough stuff to need tests you should probably be using python.

 

Standard
Coding

Terraform: Get Azure Function key

So you’ve deployed your function and you want to get pass the secure url another component in your deployment so it can use it…

Well currently there isn’t an output item on the azurerm_function_app resource in Terraform (I’m hoping to fix that up if I get some time) so how do you do it?

Here is a my quick and dirty fix using the azure_template_deployment resource in Terraform.

We create an empty release and then use the listkeys function to pull back the keys for the function. We only want the function key so we index into the object with functionKeys.default (you can get the master key too if you want).

Then we output this from the Terraform so it can be used elsewhere. You can now go ahead and pass this into your other component.

Standard