Azure Databricks and Terraform: Create a Cluster and PAT Token

[Update Feb 2021] There is now a Terraform Provider for Databricks, it’s a better route – https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs

My starting point for a recent bit of work was to try and reliably and simply deploy and manage Databricks clusters in Azure. Terraform was already in use so I set about trying to see how I could use that to also manage Databricks.

I had a look around and after trying the Terraform REST provider and a third party Datbricks provider (didn’t have much luck with either) found a Terraform Shell provider. This turned out to be exactly what I needed.

If you haven’t written a Terraform provider here’s a crash course. You basically just define a method for create, read, update and delete and the parameters they take. Then Terraform does the rest.

The Shell provider (https://github.com/scottwinkler/terraform-provider-shell) lets you do this by passing in scripts (bash, powershell, any executable that can take stdin and output stdout). In this case I wrote some powershell to wrap the databricks-cli.

It’s better (or different) to localexec with nullresources as you can store information in the Terraform State and detect drift. If a read returns different information than the current information in the state then update will be called, for example.

So I took the work of Alexandre and wrapped it into this provider and using the Shell provider have a simple, no frills Databricks provider for Terraform which makes calls to Databricks via the databricks-cli.

This is currently a simple hack and hasn’t undergone any significant testing: https://github.com/lawrencegripper/hack-databricksterraform. The flow is as follows:

Hopefully this might be useful to others as a starting point for others.

Azure, Coding, Uncategorized

Friends don’t let friends commit Terraform without fmt, linting and validation

So it starts out easy, you write a bit of terraform and all is going well then as more and more people start committing and the code is churning things start to get messy. Breaking commits block release, formatting isn’t consistent and and errors get repeated.

Seems a bit odd right, in the middle of your devops pipe which dutifully checks code passes tests and validation you just give terraform a free pass.

Captain Picard Quotes. QuotesGram

The good new is terraform has tools to help you out here and make life better!

Here is my rough script for running during build to detect and fail early on a host of terraform errors. It’s also pinning terraform to a set release (hopefully the same one you use when releasing to prod) and doing a terraform init each time to make sure you have providers pinned (if not the script fail when a provider ships breaking changes and give you an early heads up).

It’s rough and ready so make sure your happy with what it does before you give it a run. For an added bonus the docker command below the script runs it inside a Azure Devops container to emulate locally what should happen when you push.

#! /bin/bash
set -e
echo -e "\n\n>>> Installing Terraform 0.12"
# Install terraform tooling for linting terraform
wget -q https://releases.hashicorp.com/terraform/0.12.4/terraform_0.12.4_linux_amd64.zip -O /tmp/terraform.zip
sudo unzip -q -o -d /usr/local/bin/ /tmp/terraform.zip
echo ""
echo -e "\n\n>>> Install tflint (3rd party)"
wget -q https://github.com/wata727/tflint/releases/download/v0.9.1/tflint_linux_amd64.zip -O /tmp/tflint.zip
sudo unzip -q -o -d /usr/local/bin/ /tmp/tflint.zip
echo -e "\n\n>>> Terraform verion"
terraform -version
echo -e "\n\n>>> Terraform Format (if this fails use 'terraform fmt' command to resolve"
terraform fmt -recursive -diff -check
echo -e "\n\n>>> tflint"
echo -e "\n\n>>> Terraform init"
terraform init
echo -e "\n\n>>> Terraform validate"
terraform validate

view raw
hosted with ❤ by GitHub

docker run –rm -v ${PWD}:/source mcr.microsoft.com/azure-pipelines/vsts-agent:ubuntu-16.04-docker-18.06.1-ce-standard \

view raw
hosted with ❤ by GitHub

Optionally you can add args like -var java_functions_zip_file=something  to the terraform validate call.

Hope this helps as a quick rough guide!


Terraform: Get Azure Function key

Update 12/11/2020: This is now supported directly in the Azure Terraform Provider see here.

Updated 09/03/2020: This new method in the Azure provider has intermittent issues. I have another workaround here which avoids ARM templates as an alternative.

So you’ve deployed your function and you want to get pass the secure url another component in your deployment so it can use it…

Well currently there isn’t an output item on the azurerm_function_app resource in Terraform (I’m hoping to fix that up if I get some time) so how do you do it?

Here is a my quick and dirty fix using the azure_template_deployment resource in Terraform.

We create an empty release and then use the listkeys function to pull back the keys for the function. We only want the function key so we index into the object with functionKeys.default (you can get the master key too if you want).

Then we output this from the Terraform so it can be used elsewhere. You can now go ahead and pass this into your other component.

# Get the functions keys out of the app
resource "azurerm_template_deployment" "function_keys" {
name = "javafunckeys${var.random_name_ending}"
parameters = {
"functionApp" = "${azurerm_function_app.function-app.name}"
resource_group_name = "${var.resource_group_name}"
deployment_mode = "Incremental"
template_body = <<BODY
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#&quot;,
"contentVersion": "",
"parameters": {
"functionApp": {"type": "string", "defaultValue": ""}
"variables": {
"functionAppId": "[resourceId('Microsoft.Web/sites', parameters('functionApp'))]"
"resources": [
"outputs": {
"functionkey": {
"type": "string",
"value": "[listkeys(concat(variables('functionAppId'), '/host/default'), '2018-11-01').functionKeys.default]" }
output "func_keys" {
value = "${lookup(azurerm_template_deployment.function_keys.outputs, "functionkey")}"

view raw
hosted with ❤ by GitHub

// This is the response from the `listkeys` function in ARM so we use `.functionKeys.default` to reach into it and output
// the function key we need.
"functionKeys": {
"default": "KEYEHERE…qtocq1safFGhAwZkzPe1VdRflvg=="
"masterKey": "KEYEHERE……Ju1384KHUprI01kH5GIKH2uvrqew==",
"systemKeys": {}

view raw
hosted with ❤ by GitHub