Warning: This is a bit of a brain dump.
I’m working on a project at the moment which dynamically creates a set of CRDs in Kubernetes and an operator to manage them based off a schema which is provided by the user at runtime.
When the code is being built it doesn’t know the schema/shape of the CRDs. This means the standard approach used in Kubebuilder with
controller-gen isn’t going to work.
Now, for those that haven’t played with Kubebuilder it’s gives you a few super useful things to build a K8s operator in Go:
- Controller-gen creates all the structs, templated controllers and keeps all those type files in sync for you. So you change a CRDs Struct and the CRD Yaml spec is updated etc. These are all build time tools so we can’t use em.
- A nice abstraction around how to interact with K8s as a controller – The controller-runtime. As the name suggests we can use this one at runtime.
So while we can’t use the build time
controller-gen we can still use all the goodness of the
controller-runtime. In theory.
This is where the fun came in, there aren’t any docs on interacting with a dynamic/unstructured object type using the controller runtime so I did a bit of playing around.
(Code samples for illustration – if you want end2end running example skip to the bottom).
To get started on this journey we need a helping hand. Kuberentes has an API for working which objects which don’t have a Golang struct defined. This is how we can start: Lets check out the go docs for
Ok so this gives us some nice ways to work with a CRD which doesn’t have a struct defined.
To use this meaningfully we’re going to have to tell it the type it represents – In K8s this means telling it it’s
Kind. These are all wrapped up nicely in the
schema.GroupVersionKind struct. Lets look at the docs:
Great so hooking these two up together we can create an
Unstructured instance that represents a CRD, like so!
Cool, so what can we do from here? Well the controller runtime uses the
runtime.object interface for all it’s interactions and guess what we have now? Yup a
runtime.Object.. wrapper method to make things obvious
Well now we can create an instance of the controller for our
Notice that I’m passing the
GroupVersionKind into the controller struct – this will be useful when we come to make changes to a CRD we’re handling.
In the same way that you can use the
r.Client on the controller in Kubebuilder you can now use it with the
unstructured resource. We use the
gvk again here to set the type so that the
client knows how to work with it.
Now you might be thinking – wow isn’t it going to be painful working without the strongly typed CRD structs?
Yes it’s harder but there are some helper methods in the
unstructured api which make things much easier. For example, the following let you easily retrieve or set a string which is nested inside it.
unstructured.NestedString(resource.Object, "status", "_tfoperator", "tfState")
unstructured.SetNestedField(resource.Object, string(serializedState), "status", "_tfoperator", "tfState")
Here is the end result hooking up the controller runtime to a set of dynamically created and managed CRDS. It’s very much a work in progress and I’d love feedback if there are easier ways to tackle this or things that I’ve got wrong.