Quick-post

Debugging Cloud-Init on Ubuntu (in Azure or anywhere)

I’ve recently been working with cloud-init in Azure to setup Ubuntu machines and for the most part I’ve really like it as it solves lots of problems and fits my use case BUT debugging it has been a pain so I thought I’d write up some notes here for others.

Did it work?

After deployment SSH onto the node and run this:

cloud-init status --long

Did the stuff I expect get onto the node?

So things didn’t go your way, it looks like it hasn’t behaved or your just curious. Well this lets you see the contents of the ​cloud-initas it landed on your box.

sudo cat /var/lib/cloud/instance/user-data.txt.i

What exactly failed? I want the verbose logs

So the script didn’t run or something else failed and you’ve got some logging that writes to console in there, never fear this will show you what happened.

sudo cat /var/log/cloud-init-output.log

I’d like to know early if things are broken, can I validate these things on my dev machine?

Yup, this will pick up some errors but be warned the validation is limited.

  1. Get the tooling (use docker run -it  ubuntu if not on ubuntu): `sudo apt install cloud-init`
  2. Run the validation: cloud-init devel schema --config-file your-cloud-init.txt
  3. Profit

 

Note

I’m trying a new format for shorter slightly rougher blog posts covering specific topics quickly. They’ll appear under Quick-post tags. Please excuse typos and grammar issues!

Standard
Coding, Quick-post

Docker and Healthchecks outside of Kubernetes

So I’ve been working with a containerized solution recently which runs outside of Kuberenetes using an Azure VMSS to scale out. I won’t dive into the reasons why we went down this route but one really interesting thing came of out of it.

How do you automatically healthcheck a container outside of Kubernetes?

Well it turns out docker has this covered in newer versions. You can specify a HEALTHCHECK inside the docker file to monitor the containers state

How do you ensure it restarts when unhealthy?

Well here you have a couple of options but both rely on using --restart=always when starting the container:

  1. You `healthcheck` command runs inside the container so you can have it kill the root process of the container causing the container to restart – Example: https://github.com/opencb/opencga/pull/1121/files
  2. You can use `AutoHeal` container which monitors the docker deamon via it’s socket and handles and containers which report unhealthy https://hub.docker.com/r/willfarrell/autoheal/

Note: I’m trying a new format for shorter slightly rougher blog posts covering specific topics quickly. They’ll appear under Quick-post tags. Please excuse typos and grammar issues!

Standard
Azure, How to, kubernetes

Kubernetes Integration Testing: MiniKube + Azure Pipelines = Happy

I recently did some work on a fairly simple controller to run inside Kubernetes. It connects to the K8s API and watches for changes to ingress objects in the cluster.

I had a nice cluster spun up for testing which I could tweak and poke then observe the results. This was nice BUT I wanted to translate it into something that ran as part of my CI process to make it more repeatable. Having not played much with the new Azure Pipelines I decided to try and get this working using one.

Here was the goal:

    • Build the source for the controller
    • Spin up a Kuberentes cluster
    • Deploy test resources (Ingress and Services) into the cluster
    • Connect the controller code to the cluster and run it’s tests

The obvious choice was to look at creating the clusters inside a cloud provider and using it for testing but I wanted each PR/Branch to be validated independently in a separate cluster, ideally in parallel, so things get complicated and expensive if we go down that route.

Instead I worked with MiniKube which has a ‘no vm mode’, this spins up a whole cluster using just docker containers. The theory was, if the CI supports running docker containers it should support MiniKube clusters…

TLDR: Yes this is possible with MiniKube and Azure Pipelines or Travis CI – Skip to the end to see how.

Continue reading →

Standard