I’m working on a project which uses pester tests to validate our deployment and system health.
We’ve accumulated a lot of tests. Nearly all of these sit and wait on things to happen in the deployment and we’re running them sequentially which takes around 20mins. Each of our tests is in it’s own file.
What I wanted to do was, as these are IO bound tests waiting on things to trip in the environment or polling http endpoints, run them all in parallel. One file per thead or something similar.
This issue discusses this topic. I took a lead from the Invoke-Parallel snipped and started playing, unfortunately the output from the tests was mangled and overlapping.
Then I realised I could use Powershell Jobs to schedule the work and poll for the job to be completed then receive each jobs to have to output displayed nicely.
So now the output looks good:
Note: We’re using Pester v4 you’ll have to do a little bit of fiddling to port to Pester 5
What is the the aim? I have a file called IMAGETAG.txt which contains a simple version v1.0.1. It is used to build and push a Docker container as part of the build. If the file is changed in a commit, I want to build and push the docker image.
Now, for those that haven’t played with Kubebuilder it’s gives you a few super useful things to build a K8s operator in Go:
Controller-gen creates all the structs, templated controllers and keeps all those type files in sync for you. So you change a CRDs Struct and the CRD Yaml spec is updated etc. These are all build time tools so we can’t use em.
A nice abstraction around how to interact with K8s as a controller – The controller-runtime. As the name suggests we can use this one at runtime.
So while we can’t use the build time controller-gen we can still use all the goodness of the controller-runtime. In theory.
This is where the fun came in, there aren’t any docs on interacting with a dynamic/unstructured object type using the controller runtime so I did a bit of playing around.
(Code samples for illustration – if you want end2end running example skip to the bottom).
To get started on this journey we need a helping hand. Kuberentes has an API for working which objects which don’t have a Golang struct defined. This is how we can start: Lets check out the go docs for unstructured..
Ok so this gives us some nice ways to work with a CRD which doesn’t have a struct defined.
To use this meaningfully we’re going to have to tell it the type it represents – In K8s this means telling it it’s Group, Version and Kind. These are all wrapped up nicely in the schema.GroupVersionKind struct. Lets look at the docs:
Great so hooking these two up together we can create an Unstructured instance that represents a CRD, like so!
Cool, so what can we do from here? Well the controller runtime uses the runtime.object interface for all it’s interactions and guess what we have now? Yup a runtime.Object.. wrapper method to make things obvious
Well now we can create an instance of the controller for our unstructured CRD.
Notice that I’m passing the GroupVersionKind into the controller struct – this will be useful when we come to make changes to a CRD we’re handling.
In the same way that you can use the r.Client on the controller in Kubebuilder you can now use it with the unstructured resource. We use the gvk again here to set the type so that the client knows how to work with it.
Now you might be thinking – wow isn’t it going to be painful working without the strongly typed CRD structs?
Yes it’s harder but there are some helper methods in the unstructured api which make things much easier. For example, the following let you easily retrieve or set a string which is nested inside it.