#terraform, Coding, vscode

Terraform, Docker, Ubuntu 20.04, Go 1.14 and MemLock: Down the rabbit hole

I recently upgrade my machine and and installed the latest Ubuntu 20.04 as part of that.

Very smugly I fired it up the new install and, as I use devcontainers, looked forward to not installing lots of devtools as the Dockerfile in each project had all the tooling needed for VSCode to spin up and get going.

Sadly it wasn’t that smooth. After spinning up a project which uses terraform I found an odd message when running terraform plan

failed to retrieve schema from provider “random”: rpc error: code = Unavailable desc = connection error: desc = “transport: authentication handshake failed: EOF

error from terraform plan

Terraform has a provider model which uses GRPC to talk between the CLI and the individual providers. Random is one of the HashiCorp made providers so it’s a really odd one to see a bug in.

Initially I assumed that the downloaded provider was corrupted. Nope, clearing the download and retrying didn’t help.

So assuming I’d messed something up I:

  1. Tried changing the docker image using by the devcontainer. Nope. Same problem.
  2. Different versions of terraform. Nope. Same problem.
  3. Updated the Docker version I was using. Nope. Same problem.
  4. Restarted the machine. Nope. Same problem.

Now feeling quite frustrated I finally remembered a trick I’d used lots when building my own terraform providers. I enabled debug logging on the terraform CLI.

TF_LOG=DEBUG terraform plan

This is where it gets interesting…

Continue reading
Standard
Coding, Quick-post

Docker and Healthchecks outside of Kubernetes

So I’ve been working with a containerized solution recently which runs outside of Kuberenetes using an Azure VMSS to scale out. I won’t dive into the reasons why we went down this route but one really interesting thing came of out of it.

How do you automatically healthcheck a container outside of Kubernetes?

Well it turns out docker has this covered in newer versions. You can specify a HEALTHCHECK inside the docker file to monitor the containers state

How do you ensure it restarts when unhealthy?

Well here you have a couple of options but both rely on using --restart=always when starting the container:

  1. You `healthcheck` command runs inside the container so you can have it kill the root process of the container causing the container to restart – Example: https://github.com/opencb/opencga/pull/1121/files
  2. You can use `AutoHeal` container which monitors the docker deamon via it’s socket and handles and containers which report unhealthy https://hub.docker.com/r/willfarrell/autoheal/

Note: I’m trying a new format for shorter slightly rougher blog posts covering specific topics quickly. They’ll appear under Quick-post tags. Please excuse typos and grammar issues!

Standard
Azure

Integration testing Service Fabric & Traefik with Docker

Here is the plan:

  1. Use docker to run a Service Fabric Linux cluster in a container
  2. Deploy a test app to the cluster and create 25 instances of it

Aim: While developing the Traefik SF integration it will provide a simple cluster to use, debug and perform integration testing.

*TLDR: Have a look the full code in this PR

It was a fun journey but I got it working…

Continue reading

Standard