Creating a Basic Kubernetes Mutating Webhook Using Terraform
Kubernetes has a lot of superpowers, and one of the best is extendibility. One of these great extensions is a dynamic admission controller
Apr 27, 2023 • 3 Minute Read
So we know one thing’s for certain: Kubernetes is one of the most popular and powerful platforms out there right now. One of its super powers? Extendability. One of these great extensions is an admission controller — more specifically, a dynamic admission controller. They allow us to add custom logic that will figure out what and if an object like a network or compute resource for instance, can be created by meeting specific policies we define.
So, let’s take a closer look at how this works, and what and when it happens.
Here, we are creating a dynamic admission controller that will use a mutating webhook. How this works is when you create a resource like a pod, the Kubernetes API server will take a look at your webhook configuration to see if there are any admission controls that it needs to apply. If any match the operation, it applies either the mutating or validating webhooks. Our example will focus on the mutating webhook. The code for this example can be found here.
In our example, our admission control webhook is not really much more than a basic web server, as you can see in the Go code for our webhook below:
func runWebhookServer(certFile, keyFile string) {
cert, err := tls.LoadX509KeyPair(certFile, keyFile)
if err != nil {
panic(err)
}
fmt.Println("Starting webhook server")
http.HandleFunc("/mutate", mutatePod)
server := http.Server{
Addr: fmt.Sprintf(":%d", port),
TLSConfig: &tls.Config{
Certificates: []tls.Certificate{cert},
},
ErrorLog: logger,
}
if err := server.ListenAndServeTLS("", ""); err != nil {
panic(err)
}
}
Above, we are just creating a HTTP server that will listen for the /mutate path. The handler for this is quite long (so feel free to take a look at the code) but the gist of it is that we get the AdmissionReview from the request, and then do a quick test on the labels that the pod has. Basically, this sample of the code is the key part:
if _, ok := pod.Labels["hello"]; !ok { patch = `[{"op":"add","path":"/metadata/labels","value":{"hello":"world"}}]` }
So if a pod has a label with the key hello, then there will be no action. If the pod does NOT have this label, then it will create a JSON patch that adds a new label hello=world.
To create this webhook in the Kubernetes cluster, we will first need a Dockerfile. Below is an example of what that will look like:
Dockerfile
FROM ubuntu:focal
WORKDIR /opt
COPY ./bin/mutating-webhook .
CMD ["./mutating-webhook", "--tls-cert", "/etc/opt/tls.crt", "--tls-key", "/etc/opt/tls.key"]
Once that is created, we will then need to create the Terraform configuration that will deploy our webhook to our cluster. Below, we will break down the webhook_deployment.tf file.
In the first part of the configuration file we define our required providers as well as the provider configuration:
terraform {
required_providers {
kubernetes = {
source = "hashicorp/kubernetes"
}
}
}
provider "kubernetes" {
config_path = "~/.kube/config"
}
Next we will then define our mutating webhook deployment:
resource "kubernetes_deployment" "mutating_webhook" {
metadata {
name = "mutating-webhook"
}
spec {
replicas = 1
selector {
match_labels = {
app = "mutating-webhook"
}
}
template {
metadata {
labels = {
app = "mutating-webhook"
}
}
spec {
volume {
name = "cert"
secret {
secret_name = "server-cert"
}
}
container {
name = "mutating-webhook"
image = "<USERNAME>/mutating-webhook:latest"
port {
container_port = 443
}
volume_mount {
name = "cert"
read_only = true
mount_path = "/etc/opt"
}
image_pull_policy = "Always"
}
}
}
}
}
As you can see in this deployment we are pulling in the docker image of our webbook we created with our Go code. Our webhook will use port 443, so TLS certs are required.
Our next resource that is being defined is the service that points to our mutating webhook. It should look like the example below:
resource "kubernetes_service" "mutating_webhook" {
metadata {
name = "mutating-webhook"
}
spec {
port {
port = 443
target_port = "443"
}
selector = {
app = "mutating-webhook"
}
}
}
And finally the last resource that is being defined is for the MutatingWebhookConfiguration which tells the Kubernetes API how and when to call our webhook:
resource "kubernetes_mutating_webhook_configuration" "pod_label_add" {
depends_on = [kubernetes_deployment.mutating_webhook]
metadata {
name = "pod-label-add"
annotations = {
"cert-manager.io/inject-ca-from" = "default/client"
}
}
webhook {
name = "pod-label-add.guru.com"
client_config {
service {
namespace = "default"
name = "mutating-webhook"
path = "/mutate"
}
}
rule {
api_groups = [""]
api_versions = ["v1"]
resources = ["pods"]
operations = ["CREATE"]
scope = "Namespaced"
}
side_effects = "None"
admission_review_versions = ["v1"]
}
}
This piece of configuration defines the clientConfig, which points to the mutating-webhook Kubernetes service that we created. It also defines that the API server should use the /mutate route that we specified in our webhook server. Then, near the bottom of the configuration, we create the rules on which resources to apply this mutating webhook to. In our example, the mutating webhook should apply to all pods.
To see how this works, we will need to create a simple pod and deploy it:.
It is best to create a new directory called something like test-pods and create this configuration within it.
terraform {
required_providers {
kubernetes = {
source = "hashicorp/kubernetes"
}
}
}
provider "kubernetes" {
config_path = "~/.kube/config"
}
resource "kubernetes_pod" "test_app" {
metadata {
name = "test-app"
}
spec {
container {
name = "test-app"
image = "ubuntu:focal"
command = ["/bin/bash"]
args = ["-c", "sleep infinity"]
}
}
}
As you can see above, we have not defined any labels for the pod.
Once the pods has been deployed and created, we can take a better look at it using the describe command:
$ kubectl describe po test-app
Name: test-app
Namespace: default
Priority: 0
Node: kind-control-plane/172.18.0.3
Start Time: Sun, 14 Nov 2021 14:42:16 -0500
Labels: hello=world
Annotations: <none>
Status: Running
... and so on and so forth
It looks like our mutating webhook worked! You can see that the label hello=world was added to the pod.
Conclusion
Hopefully this has helped you understand how a basic mutating webhook for dynamic admission control works in Kubernetes, and how to implement one using Terraform.
If you would like to know more about Kubernetes Admission Controllers and how to manage and deploy them with Terraform, please have look at my course Deploying Custom Admission Controllers with Terraform.
Also some other resources that may peak your interests:
