Kustomize for composing and validating generic (non-Kubernetes) configuration

Kustomize allows us to build configuration without templates. We can define base configuration files and then transform them - for example, a typical use-case with Kubernetes might be defining a Deployment and then updating the replicas (replica count) property depending on the environment. Instead of parameterising this value, we define it in an overlay. The file structure might look like this:

1
2
3
4
5
6
7
|-- base
|   |--deployment.yaml
|   |--kustomization.yaml
|--overlays
   |--production
   |  |--replica_count.yaml
   |  |--kustomization.yaml

The base deployment.yaml file would contain our Deployment definition. The overlay replica_count.yaml would contain the same kind and name metadata properties as the deployment manifest in base, but then just the properties we want to overlay. For example:

1
2
3
4
5
6
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  replicas: 5

The Kustomizaiton files define references to the resources that should be included. For the base file, it would look like this:

1
2
resources:
- deployment.yaml

For the overlay Kustomization file, it would also define how the overlay resources should be applied. There’s a few ways to do this. In this case, we can define the overlay files as patches (see docs here).

1
2
3
4
5
resources:
- ../../base

patches:
- replica_count.yaml

The important thing to note is that bases (defined as resources in the Kustomization file for the overlay) have no knowledge of overlays - a base could be used in multiple overlays. We can also reference bases in Git (for example, github.com/robselway/kustomizeexample/examples/base?ref=v1.0.0).

Kustomize has lots of handy features and short-cuts for common use-cases. For example, if you’re updating an image property you don’t have to define another file - you can do it inline in the overlay Kustomization file:

1
2
3
4
images:
- name: myImage
  newName: my-registry/my-image
  newTag: v2

There’s also a short-cut for updating the replica count, so defining a file like we did above isn’t strictly necessary.

For a full list of possibilities with Kustomize, see the docs here.

Installing Kustomize

The documentation outlines a few ways to install Kustomize. I’m running Ubuntu so used the go get command but changed $GOBIN to be /user/local/bin, which is already part of $PATH. This requires elevated permmissions.

1
sudo GOBIN=/usr/local/bin/ GO111MODULE=on go get sigs.k8s.io/kustomize/kustomize/v3

Handling generic configuration

Although Kustomize is intended for use with Kubernetes manifest files, it can also be used to generically compose configuration.

For example, you might have generic configuration containing data like IP addresses for allow lists, or company information such as trading name and VAT number. Kustomize allows us to compose this information together and overlay additional information depending on the tenant (for example, additional office IP addresses in production for an internal application). This configuration could then be presented as JSON files for use outside of Kubernetes.

Kustomize only requires two properties in a manifest file:

  • .kind
  • .metadata.name

As long as we have these, we can define whatever we like for the rest of the documents.

The full example is on GitHub. We start with the following file structure:

1
2
3
4
5
6
7
8
|-- base
|   |--company_info.yaml
|   |--security_config.yaml
|   |--kustomization.yaml
|--overlays
   |--production
   |  |--security_config.yaml
   |  |--kustomization.yaml

company_info.yaml looks like this:

1
2
3
4
5
6
7
kind: custom-config
metadata:
  name: company-info
config:
  trading_name: ExampleCorp
  vat_number: 123456

And security_config.yaml looks like this:

1
2
3
4
5
6
7
kind: custom-config
metadata:
  name: security-config
config:
  office_ips:
  - 1.1.1.1
  - 2.2.2.2

In this example we simply want to append some additional IP addresses to .config.office_ips. We can do this using the patchesJson6902 functionality in Kustomize:

1
2
3
4
5
6
7
8
9
10
11
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../bases

patchesJson6902:
- target:
    kind: custom-config
    name: security-config
  path: security_config_jsonpatch.yml

security_config_jsonpatch.yml looks like this:

1
2
3
4
5
6
7
- op: add
  path: "/config/office_ips/-"
  value: 4.4.4.4

- op: add
  path: "/config/office_ips/-"
  value: 5.5.5.5

And finally, the output of kustomize build overlays/production:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
config:
  trading_name: ExampleCorp
  vat_number: 123456
kind: custom-config
metadata:
  name: company-info
---
config:
  office_ips:
  - 1.1.1.1
  - 2.2.2.2
  - 4.4.4.4
  - 5.5.5.5
kind: custom-config
metadata:
  name: security-config

Splitting the files and converting to JSON

For my use case, I want the end result to be separate JSON files for each document. I can accomplish this with a bash script (thanks to this StackOverlow answer for the inspiration):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# Version 4.11.2 of yq (https://github.com/mikefarah/yq)
# Tidy up
mkdir -p .outputs
rm -rf .outputs/*

# Config for Kustomize to discover plugins in right place
export XDG_CONFIG_HOME=$(pwd)

# Split YAML documents into individual JSON files
# Script adapted from https://stackoverflow.com/a/67876873
# --enable_alpha_plugins required to allow custom plugin
x="$(kustomize build --enable_alpha_plugins overlays/production)"
while [[ -n "$x" ]]
do ary+=("${x%%---*}")
if [[ "$x" =~ --- ]]; then x="${x#*---}"; else x=''; fi
done
for yml in "${ary[@]}";
    do fileName=`echo "$yml" | yq eval '.metadata.name' -`;
    echo $(echo "$yml" | yq eval -j) > .outputs/$fileName.json;
done

# Remove intermediate file
rm -rf kustomize_output.yml

This leaves me with the following output:

1
2
3
|-- .outputs
|   |--company-info.json
|   |--security-config.json

Validation

Kustomize provies a structured way to validate the output of our configuration. We can create a validator plugin (a bash script in this case) and reference it using a standard YAML manifest file. The file structure we had above will have the following added to it:

1
2
3
4
5
6
7
8
9
|-- kustomize
|   |-- plugin
|   |   |-- myteam.example.com
|   |   |   |-- v1
|   |   |   |   |-- validator
|   |   |   |   |   |-- Validator
|--overlays
   |--production
   |  |--validator.yaml

The validator.yaml looks like this (the apiVersion is deconstructed to resolve to the correct file path - the .metadata.name property does not matter):

1
2
3
4
apiVersion: security.example.com/v1
kind: Validator
metadata:
  name: does-not-matter

This file is referenced in the Kustomization file like so:

1
2
validators:
- validator.yaml

In this validator I want to check that a particular IP address is always included in our security-config file, in the .config.office_ips list. The script looks like this (inspired from the same StackOverflow answer):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#!/bin/bash

# Read stdin into variable
x=`cat -`

# Split YAML documents into individual JSON files
# Script adapted from https://stackoverflow.com/a/67876873
while [[ -n "$x" ]]
do ary+=("${x%%---*}")
if [[ "$x" =~ --- ]]; then x="${x#*---}"; else x=''; fi
done
for yml in "${ary[@]}";
do
    # Find name of document and check it's what we are looking for
    fileName=`echo "$yml" | yq eval '.metadata.name' -`;
    if [ $fileName = 'security-config' ]
    then
        # Find the IP address we want to check exists
        ipResult=`echo "$yml" | yq eval '.config.office_ips.[] | select(. == "4.4.4.4")' -`

        # Exit if result is empty
        if [[ ! $ipResult ]]
        then
            >&2 echo "Important IP addresses missing from config: 4.4.4.4"
            exit 1
        fi
    fi
done

The script must not alter the output (Kustomize will check and throw an error if this happens). There is one exception - a label of validated-by can be added.

We also need to use the --enable_alpha_plugins flag when calling kustomize build, otherwise you will see an error like this:

1
Error: external plugins disabled; unable to load external plugin

On a successful validation, the output proceeds as normal. When validation fails, we get a friendly error messasge:

1
2
Important IP addresses missing from config: 4.4.4.4
Error: failure in plugin configured via /tmp/kust-plugin-config-218317647; exit status 1: exit status 1

Conclusion

Kustomize offers a structured and declarative way of composing configuration. As it only requires minimal metadata in each YAML file to work, we can use it for any configuration schema we like.

Verifying the configuration is an added bonus - although it’s worth thinking through any security and stability implications of allowing the tool to run a script in your environment.

The validation I demonstrated here is relatively simple. If you are composing Kubernetes manifests, you could use a tool like kubeval to check the files against the K8s OpenAPI schemas.

Further reading