First up, quick refresher – what is a mutating admission controller?
Well it’s a nice feature in Kubernetes which lets you intercept objects when they’re created and make changes to them before they are deployed into the cluster.
Cool right? All those fiddly bits of YAML or hard to enforce company policies around network access, image stores you can and can’t use, they can all be enforced and FIXED automagically! (Like all magic caution is advised, choose wisely – queue Monty python gif)
So what’s the catch? Well without Open Policy Agent (OPA) you had to build out a web api to do the magic of changing the object then build/push an image and go through maintaining the solution. While you can write them quite easily now with solutions like KubeBuilder, or if you really love node I build one using that too, I wanted to see if OPA made things easier.
So say you want something more dynamic, flexible and a little easier to look after?
Today I’ve been having a play with it to work out if I could build a controller which would set a certain nodeSelector on pods based on which namespace they are deployed in.
I’ll go over this very broadly I highly recommend looking at the docs in detail before diving in, I lost quite a bit of time to not reading things properly before starting.
I won’t lie, getting used to the DSL (rego) was painful for me, mainly because I came at it thinking it was going to be really like Golang. It does look quite like it but that’s where the similarity ends, it’s more functional/pattern matching and better suited to tersely making decisions based on data.
To counter the learning curve of rego I have to say, as I’ve raised issues and contributions the maintainers have been super responsive and helpful (even when I’ve made some silly mistakes) and the docs are great with runnable samples to get started.
Lets talk more about what I built out.
Health warning: I’m new to this and still learning, I may get some of this wrong. Over the next few days I’ll be doing more testing and loop back to fix things up.
First up you need to process the input from the request input and output a main object which will be the response sent to the K8s API.
The first response on line 10 is the default which is returned if nothing else takes over.
The second response is a mix between and object definition and a set of rules lines 18->25 (think assertions). If the rules all match then the assignment takes place and response becomes the output object on line 49. After the assertions there is some plumbing which builds up the response with the JSON patches K8s is looking, these set the nodeselector
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # The top level response sent to the webrequest | |
| main = { | |
| "apiVersion": "admission.k8s.io/v1beta1", | |
| "kind": "AdmissionReview", | |
| "response": response, | |
| } | |
| # If the conditions on the `response` below aren't met this default `allow` response | |
| # is returned. | |
| default response = {"allowed": true } | |
| # This is the response body sent back to admissions controller request | |
| # it starts with a number of conditions which have to be met for it to take effect | |
| # | |
| # Note: output is the return item, inside the body the `output :=` sets this so that response equals | |
| # the value of the `output` object defined in this body. | |
| response = output { | |
| # Retrun this if request is: | |
| # | |
| # A pod | |
| isPod | |
| # Without any pre-existing selectors | |
| not hasNodeSelector | |
| # And in a namespace we care about | |
| shouldProcessForNamespace(ignoredNamespaces) | |
| # Generate the JSON Patch object | |
| patch := { | |
| "op": "add", | |
| "path": "spec/nodeSelector", | |
| "value": { | |
| # Retrieve the `pool` name which should be applied given the | |
| # namespace in which this pod is created. | |
| "agentpool": getPoolForNamespace(input.request.object.metadata.namespace) | |
| } | |
| } | |
| # Patches have to be an array of base64 encoded JSON Patches so lets | |
| # make our single patch into an array, serialize as JSON and base64 encode. | |
| patches := [patch] | |
| patchEncoded := base64.encode(json.marshal(patches)) | |
| # Output a trace use `opa test *.rego -v –explain full` to see them. | |
| trace(sprintf("POLICY:generatedPatch raw = '%s'", [patches])) | |
| trace(sprintf("POLICY:generatedPatch encoded = '%s'", [patchEncoded])) | |
| # Generate the patch response and return it! We're done! | |
| output := { | |
| "allowed": true, | |
| "patchType": "JSONPatch", | |
| "patch": patchEncoded | |
| } | |
| } |
So how do these rules work then? Well they’re like little functions that check certain things.
In isPod we do a simple equality check on the kind of the request.
In hasNodeSelector we check if the pod already has a node selector by first checking if the field exists, if it doesn’t the next check doesn’t happen, then checking how many items are in it.
getPoolForNamespace is a special case it takes an input namespace then loops through the array defined as namespaceToAgentPool and sees if any pools match the namespace of the pod. The magic happens here with the namespaceToAgentPool[_] which means, roughly, check the rules against all of the items in that array.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Rule: Check if the item submitted is a pod. | |
| isPod { | |
| input.request.kind.kind == "Pod" | |
| } | |
| # Rule: Check if pod already has a `nodeSelector` set | |
| hasNodeSelector { | |
| input.request.object.spec.nodeSelector | |
| count(input.request.object.spec.nodeSelector) > 0 | |
| } | |
| # Rule: Given a namespace iterate through the `namespaceToAgentPool` array | |
| # and return the value which the `agentpool` should be set to in the | |
| # node selector. | |
| getPoolForNamespace(namespace) = poolLabel { | |
| pool := namespaceToAgentPool[_] | |
| pool.namespace == namespace | |
| poolLabel := pool.agentpool | |
| } | |
| # Rule: Checks if the object is in a namespace we should process. | |
| shouldProcessForNamespace(ignored) { | |
| not contains(ignored, input.request.object.metadata.namespace) | |
| } | |
| # Rule: Helper to check if an array contains an instance of `item` | |
| contains(items, item) { | |
| items[_] == item | |
| } | |
| # Data: Used to map namespace -> agentpools… | |
| # Would be updated with more rules as the list grows | |
| namespaceToAgentPool := [ | |
| { "namespace": "default", "agentpool": "pool1"}, | |
| { "namespace": "gpuwork", "agentpool": "gpu1"}, | |
| { "namespace": "memintensivework", "agentpool": "highmem1"}, | |
| ] | |
| # Data: Namespaces which we should ignore when processing requests | |
| # so we don't mess with any system pods etc. | |
| # todo// check not missing any | |
| ignoredNamespaces := [ | |
| "kube-node-lease", | |
| "kube-public", | |
| "kube-system", | |
| "opa" | |
| ] |
So how do you know this stuff works? Well it’s got a nice testing framework you can use to check things are working how you expect. I found the trace command super useful when running opa test *.rego -v --explain full as it would prove the value of the items passed to it along with other information about the execution.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # patch we expect to be generated | |
| expectedPatch = { | |
| "op": "add", | |
| "path": "spec/nodeSelector", | |
| "value": { | |
| "agentpool": "pool1" | |
| } | |
| } | |
| # Helper to check patch is set | |
| hasPatch(patches, expectedPatch) { | |
| # One of the patches returned should match the `expectedPatch` | |
| patches[_] == expectedPatch | |
| } | |
| # Checks the response is a patch response | |
| isPatchResponse(res) { | |
| # Is the response patch type correct? | |
| res.response.patchType = "JSONPatch" | |
| # Is the patch body set? | |
| res.response.patch | |
| # Is the patch body an array with more than one item? | |
| count(res.response.patch) > 0 | |
| } | |
| # Test that the controller correctly sets a patch on a pod | |
| # to assign it the correct `nodeSelector` of `agentpool=pool1` | |
| test_response_patch { | |
| # Invoke the policy main with a pod which doesn't have a node selector | |
| # and is in the default namespace | |
| body := main with input as testdata.example_pod_doesnt_have_node_selector | |
| # Check policy returned an allowed response | |
| body.response.allowed = true | |
| # Check the response is a patch response | |
| isPatchResponse(body) | |
| # The admission controller response is an array of base64 encoded | |
| # jsonpatches so deserialize so we can review them. | |
| patches := json.unmarshal(base64.decode(body.response.patch)) | |
| # Output some tracing… `opa test *.rego -v –explain full` to see them | |
| trace(sprintf("TEST:appliedPatch = '%s'", [patches])) | |
| trace(sprintf("TEST:expectedPatch = '%s'", [expectedPatch])) | |
| # Check the policy created the expected patch | |
| hasPatch(patches, expectedPatch) | |
| } |
So all together now this looks like: https://github.com/open-policy-agent/contrib/pull/93
Hopefully this is useful. For more advanced stuff there is a library of shared helpers that can be pulled into rego which is well worth a look: https://github.com/open-policy-agent/library/tree/master/kubernetes/mutating-admission nearly everything I’ve done here is largely based on simplifying those funcs and adding more comments.
GripDev 