Uncategorized

Using Azure DevOps to speed up Docker builds

[Braindump – warning]

So I’ve been playing with devcontainers for Visual Studio Code, they’re awesome… go play with them. They let you use a Dockerfile to describe all the tooling needed for devs to get started with your project.

One of the side effects is that you have a nice Dockerfile which you can then also use it for your build server meaning that you never have an inconsistency between your local setup and your CI server.

In this example I build a golang project and use Azure DevOps and use caching to minimize the amount of time for each build.

trigger:
master
pool:
vmImage: 'Ubuntu-16.04'
steps:
# Cache the docker image file
task: CacheBeta@0
inputs:
key: go-cache | go.sum
path: ".gocache"
restoreKeys: go-cache
displayName: Cache go mod cache
# Cache the docker image file
task: CacheBeta@0
inputs:
key: docker-image | .devcontainer/**
path: ".dockercache"
restoreKeys: docker-image
cacheHitVar: DOCKER_CACHE_HIT
displayName: Cache docker layers
script: |
bash -f ./ci.sh
displayName: 'Run CI'

view raw
azure-pipelines.yaml
hosted with ❤ by GitHub

#! /bin/bash
set -e
set -x
# Get storage drive details
docker info
# Create .dockercache directory
mkdir -p ./.dockercache/
# Import devcontainer from cache to speed up build
if [ -f ".dockercache/devcontainer.tar" ];
then
echo "——-> Restoring docker image"
time docker load -i .dockercache/devcontainer.tar
fi
echo "——-> Building devcontainer"
# Use the devcontainer to run the build as it has all the environment setup that we need
time docker build –cache-from devcontainer:latest -t devcontainer -f ./.devcontainer/Dockerfile ./.devcontainer
# Create a directory for go mod cache
mkdir -p ${PWD}/.gocache
echo "——-> Building code"
# Run `make` to build and test the code
time docker run -v ${PWD}/.gocache:/go/pkg/ -v /var/run/docker.sock:/var/run/docker.sock -v ${PWD}:/src –workdir /src –entrypoint /bin/bash devcontainer -c "make"
# Ensure .gocache permmissions correct for build to save cache
sudo chown -R $USER ./.gocache
# If the current cached image is out of date save devcontainer so it can be cached
if [ $DOCKER_CACHE_HIT != "true" ];
then
echo "——-> Saving docker image"
time docker image save -o ./.dockercache/devcontainer.tar devcontainer
fi

view raw
ci.sh
hosted with ❤ by GitHub

Standard
Uncategorized

Misusing C# 8 `using` to get Golang `defer` in DotNet Core 3

I started out with C# since then I’ve learned other languages and one of my favorites is Golang.

When I was reading the release notes from C# 8 I saw the new using declaration and through it was awesome… I also realized it could be misused to give C# the defer keyword from Golang.

Whats defer in Golang do?

defer in Golang lets you define a function that will get run when the code block exits.

The example below will print:

hello
world

Code: Try it out here


package main

import "fmt"

func main() {
    defer fmt.Println("world")
    fmt.Println("hello")
}

I’m a big fan of this approach as I think it’s clearer to read than the standard try finally pattern as it lets you put the cleanup code directly below the code which is making the mess like:

    f := createTempFile("/tmp/defer.txt")
    defer deleteTempFile(f)
    writeFile(f)
    
    // here the `deleteTempFile` method gets called

How can we do this in C# 8 with the new using syntax?

Well now because using var x = new thing()exists you can write a simple class called defer which runs a function when the current method exits, just like in golang

The interesting thing is that as a using statement generates a try finally under the covers these defer functions will still run if an exception is thrown.

The example below will print:

Hello World!
Defer 2
Defer 1
Exception thrown: It's all one wrong

Code:

using System;
namespace deferdotnet
{
class Program
{
static void Main(string[] args)
{
try
{
DoStuff();
}
catch (Exception e)
{
Console.WriteLine($"Exception thrown: {e.Message}");
}
}
private static void DoStuff()
{
using var d = new Defer(() => Console.WriteLine("Defer 1"));
using var f = new Defer(() => Console.WriteLine("Defer 2"));
Console.WriteLine("Hello World!");
throw new Exception("It's all one wrong");
}
}
// Do the `defer` magic
public class Defer : IDisposable
{
private Action deferFunc { get; }
public Defer(Action deferFunc)
{
this.deferFunc = deferFunc;
}
public void Dispose()
{
deferFunc();
}
}
}

view raw
program.cs
hosted with ❤ by GitHub

Should you do this? Well that’s kinda up to you, I don’t think it’s super nasty but I’ve not used this in anger so probably worth testing out a bit before going all out.

Standard
Azure, Coding, Uncategorized

Friends don’t let friends commit Terraform without fmt, linting and validation

So it starts out easy, you write a bit of terraform and all is going well then as more and more people start committing and the code is churning things start to get messy. Breaking commits block release, formatting isn’t consistent and and errors get repeated.

Seems a bit odd right, in the middle of your devops pipe which dutifully checks code passes tests and validation you just give terraform a free pass.

Captain Picard Quotes. QuotesGram

The good new is terraform has tools to help you out here and make life better!

Here is my rough script for running during build to detect and fail early on a host of terraform errors. It’s also pinning terraform to a set release (hopefully the same one you use when releasing to prod) and doing a terraform init each time to make sure you have providers pinned (if not the script fail when a provider ships breaking changes and give you an early heads up).

It’s rough and ready so make sure your happy with what it does before you give it a run. For an added bonus the docker command below the script runs it inside a Azure Devops container to emulate locally what should happen when you push.

#! /bin/bash
set -e
SCRIPT_DIR=$(dirname "$BASH_SOURCE")
cd "$SCRIPT_DIR"
echo -e "\n\n>>> Installing Terraform 0.12"
# Install terraform tooling for linting terraform
wget -q https://releases.hashicorp.com/terraform/0.12.4/terraform_0.12.4_linux_amd64.zip -O /tmp/terraform.zip
sudo unzip -q -o -d /usr/local/bin/ /tmp/terraform.zip
echo ""
echo -e "\n\n>>> Install tflint (3rd party)"
wget -q https://github.com/wata727/tflint/releases/download/v0.9.1/tflint_linux_amd64.zip -O /tmp/tflint.zip
sudo unzip -q -o -d /usr/local/bin/ /tmp/tflint.zip
echo -e "\n\n>>> Terraform verion"
terraform -version
echo -e "\n\n>>> Terraform Format (if this fails use 'terraform fmt' command to resolve"
terraform fmt -recursive -diff -check
echo -e "\n\n>>> tflint"
tflint
echo -e "\n\n>>> Terraform init"
terraform init
echo -e "\n\n>>> Terraform validate"
terraform validate

view raw
validate-tf.sh
hosted with ❤ by GitHub

docker run –rm -v ${PWD}:/source mcr.microsoft.com/azure-pipelines/vsts-agent:ubuntu-16.04-docker-18.06.1-ce-standard \
/source/deployment/validate-tf.sh

view raw
zdocker-local.sh
hosted with ❤ by GitHub

Optionally you can add args like -var java_functions_zip_file=something  to the terraform validate call.

Hope this helps as a quick rough guide!

Standard