Coding, Quick-post

Avoiding pushing secret stuff to Git by accident

So it seems like a brain dead simple one. Don’t push secrets by accident, make sure you check and update the projects .​gitignore to ignore sensitive files but the reality is different.

One example, you use Terraform and set the ignore file to ignore the state file. Then later another developer moves the folder the Terraform is in and updates the ignore. Now when you merge you get the updated .gitignore and if you don’t pay attention all your state files get pushed in your next commit.

Whats the solution?

Global Git Ignores! Yes they exist and are easy to use. Check out this guide

So using this you can setup a nice rule like this:

*.private*
private.*

Now next time you create a file you NEVER want to end up in a commit all you have to do is name it secretstuff.private.env and your safe.

It’s saved me loads and I can’t recommend it enough – also you can update your global with more specific stuff like Terraform or whatever else you want.

Standard
Quick-post

Debugging Cloud-Init on Ubuntu (in Azure or anywhere)

I’ve recently been working with cloud-init in Azure to setup Ubuntu machines and for the most part I’ve really like it as it solves lots of problems and fits my use case BUT debugging it has been a pain so I thought I’d write up some notes here for others.

Did it work?

After deployment SSH onto the node and run this:

cloud-init status --long

Did the stuff I expect get onto the node?

So things didn’t go your way, it looks like it hasn’t behaved or your just curious. Well this lets you see the contents of the ​cloud-initas it landed on your box.

sudo cat /var/lib/cloud/instance/user-data.txt.i

What exactly failed? I want the verbose logs

So the script didn’t run or something else failed and you’ve got some logging that writes to console in there, never fear this will show you what happened.

sudo cat /var/log/cloud-init-output.log

I’d like to know early if things are broken, can I validate these things on my dev machine?

Yup, this will pick up some errors but be warned the validation is limited.

  1. Get the tooling (use docker run -it  ubuntu if not on ubuntu): `sudo apt install cloud-init`
  2. Run the validation: cloud-init devel schema --config-file your-cloud-init.txt
  3. Profit

 

Note

I’m trying a new format for shorter slightly rougher blog posts covering specific topics quickly. They’ll appear under Quick-post tags. Please excuse typos and grammar issues!

Standard
Coding, Quick-post

Docker and Healthchecks outside of Kubernetes

So I’ve been working with a containerized solution recently which runs outside of Kuberenetes using an Azure VMSS to scale out. I won’t dive into the reasons why we went down this route but one really interesting thing came of out of it.

How do you automatically healthcheck a container outside of Kubernetes?

Well it turns out docker has this covered in newer versions. You can specify a HEALTHCHECK inside the docker file to monitor the containers state

How do you ensure it restarts when unhealthy?

Well here you have a couple of options but both rely on using --restart=always when starting the container:

  1. You `healthcheck` command runs inside the container so you can have it kill the root process of the container causing the container to restart – Example: https://github.com/opencb/opencga/pull/1121/files
  2. You can use `AutoHeal` container which monitors the docker deamon via it’s socket and handles and containers which report unhealthy https://hub.docker.com/r/willfarrell/autoheal/

Note: I’m trying a new format for shorter slightly rougher blog posts covering specific topics quickly. They’ll appear under Quick-post tags. Please excuse typos and grammar issues!

Standard