TrueK8S Part 03

Creating an encrypted ConfigMap for secure variable susbtitution with flux.

As we write yaml files for kubernetes resources, we’ll be referencing a lot of repetitive and/or sensitive information such as IPs, API Keys, or passwords. Because we’re using flux, all of that sensitive information is going to end up in a git repo. Moreover, we’re going to have to do a lot of silly copy pasting to reference common values. Copy pasting over and over sucks and storing secrets in a github repo (even a private one) is poor practice. Thankfully flux provides mechanisms to deal with this: SOPS for encrypting and decrypting secrets, and post build substitution to implement variables in our deployment resources. Using these two mechanisms we’ll create a kuberentes ConfigMap full of variables that we can reference in our deployment references.

SOPS (Secret OPerationS) is a tool that can encrypt various file formats, in our case YAML. It knows how to encrypt just the value portions of our files while leaving the rest in-tact. Flux understand SOPS and can use it to decrypt our encrypted cert-manager on the fly. We’ll provide our secret key to Flux securely, and then use the post build susbstituion configuration to perform variable substitution on our deployment resources referencing the secrets from our ConfigMap.

Configure SOPS and AGE

Let’s start by installing SOPS (the encryption tool) and age (tools for the specific encryption algorithm we will use with SOPS).

Install SOPS:

# Download the binary
curl -LO https://github.com/getsops/sops/releases/download/v3.10.2/sops-v3.10.2.linux.amd64

# Move the binary into your PATH
mv sops-v3.10.2.linux.amd64 /usr/local/bin/sops

# Make the binary executable
chmod +x /usr/local/bin/sops

and install AGE:

$ apt install age

We’ll use the age tools we just installed to create an age encryption key pair. This will generate a key.txt file including your public and private key, so make sure to save this file securely OUTSIDE of your git repo.

To create the key pair:

$ age-keygen -o sops-key.txt
Public key: age1uprm5p9crdz7xn3ra9kfwv2axyg6re0jgn6tknf9tmsrn7dm2u3qktlpgx

Inspect the contents of your key file, it should look like this:

$ cat sops-key.txt
created: 2025-04-28T18:30:27-05:00
public key: age1uprm5p9crdz7xn3ra9kfwv2axyg6re0jgn6tknf9tmsrn7dm2u3qktlpgx
AGE-SECRET-KEY-#############################################################

Now copy your key file to your ~/.config/sops/age/keys.txt so that SOPS knows where to find it. You’ll probably need to create the directories:

$ mkdir ~/.config/sops
$ mkdir ~/.config/sops/age
$ nano ~/.config/sops/age/keys.txt

Now we need to give our encryption keys to flux. We’ll create a generic secret in the flux-system namespace and import content from our sops-key.txt file.

$ kubectl -n flux-system create secret generic sops-age --from-file=age.agekey=sops-key.txt

Finally, we need to create a .sops config file within our repo to tell SOPS exactly how we want it to encrypt files within our repository. SOPS looks for a .SOPS.yaml file in any parent directory when it runs, so we’ll create that file in the config directory and populate it with our PUBLIC key (don’t put your age private key here).

config$ cat .sops.yaml 
creation_rules:
  - encrypted_regex: '^(data|stringData)$'
    age: age1uprm5p9crdz7xn3ra9kfwv2axyg6re0jgn6tknf9tmsrn7dm2u3qktlpgx

Create an encrypted ConfigMap resource in flux

Now that we have flux set to use our SOPS key to decrypt files, we’re going to create a ConfigMap and encrypt it with SOPS. If you’ve been following along, you should have a flux repo structure like this:

truek8s@debian02:~/Documents$ tree truek8s
truek8s
├── apps
│ ├── general
│ └── production
│     └── kustomization.yaml
├── clusters
│ └── truek8s
│     ├── apps.yaml
│     ├── config.yaml
│     ├── flux-system
│     │ ├── gotk-components.yaml
│     │ ├── gotk-sync.yaml
│     │ └── kustomization.yaml
│     ├── infrastructure.yaml
│     └── repos.yaml
├── config
│ ├── .sops.yaml
│ └── kustomization.yaml
── infrastructure
│ ├── core
│ └── production
│     └── kustomization.yaml
└── repos
    └── kustomization.yaml

We’re going to create our new ConfigMap in the config directory, where it will be applied by the /clusters/truek8s/config kustomization. We’ll start by making a tweak to the /clusters/truek8s/config kustomization. Go ahead and add the spec.decryption section shown below. This will instruct flux to use the secret we created earlier to decrypt any encypted files it finds. It should look like this:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: config
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./config
  prune: true
  wait: true
  decryption:
    provider: sops
    secretRef:
      name: sops-age

Next, create the config map at ./config/cluster-config.yaml, and populate it with a yaml ConfigMap definition like this. You can define any arbitrary values under the data field in the name: value format. In this case, we’ve created a ConfigMap that maps ‘test’ to ‘Value’, and we can reference ‘test’ as a variable later in other flux resources.

apiVersion: v1
kind: ConfigMap
metadata:
    name: cluster-config
    namespace: flux-system
data:
    test: Value

And don’t forget to update the ./config/kustomization.yaml so that it applies the new cluster-config.yaml file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: 
  - cluster-config.yaml

Finall, we can encrypt our ConfigMap resource and commit everything to our repo. From the ./config run this command to encrypt the file:

sops -e -i cluster-config.yaml

This will transform your ConfigMap resource from a plaintext file to a SOPS encrypted file, like this:

apiVersion: v1
kind: ConfigMap
metadata:
    name: cluster-config
    namespace: flux-system
data:
    test: ENC[AES256_GCM,data:ajzYI0s=,iv:bZ+1ZK4GnaXTkJWgqwWmVj8gPfgeBRHtDFc/kyOtbEA=,tag:fctQUsbxfyNyJ/w+/6Ob7Q==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age:
        - recipient: age1uprm5p9crdz7xn3ra9kfwv2axyg6re0jgn6tknf9tmsrn7dm2u3qktlpgx
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBSWDJITGhjQjM4YnRCQUk1
            ----------------------------REDACTED----------------------------
            +XUzewHq1uEa6yvEIu12gu7sLg5yfx93UF44GUDpG8VKWt0vqTuTXA==
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2025-05-01T00:19:39Z"
    mac: ENC[AES256_GCM,data:gFugdPOc/nyLTb8nIo4VjYcYbJSePw2zPXmn1uo5yMGhMQ3y0spwBg9jlCl/lbVzCIVeWJ2aoMI9DAGFglHTTPjNOhyJ9UOBk+Eyh3NpgF+/J9wP8oZXTrIOaNZP+L7BAZnHNgN1CR1obvckBXeta2Eei3ozWptrrs4wy0CXgfQ=,iv:9szMfPuxvt44jtT11cY8IM19DxgVMDHuKsq+tM4D+Y8=,tag:9RzRy2l3NJCQodprE9spmA==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.9.4

Go ahead and commit your changes and watch flux as it works on the commit.

git add -A && git commit -m "Added cluster-config"
git push

flux events --watch

Verify cluster-config

If all goes well, you should be able to dump the cluster-config confimap using kubectl and see the decrypted variables in the data field:

$ kubectl get configmaps -n flux-system cluster-config -o json
{
    "apiVersion": "v1",
    "data": {
        "test": "Value"
    },
    "kind": "ConfigMap",
    "metadata": {
        "creationTimestamp": "2025-04-28T23:53:14Z",
        "labels": {
            "kustomize.toolkit.fluxcd.io/name": "kubernetes",
            "kustomize.toolkit.fluxcd.io/namespace": "flux-system"
        },
        "managedFields": [
            {
                "apiVersion": "v1",
                "fieldsType": "FieldsV1",
                "fieldsV1": {
                    "f:data": {
                        "f:test": {}
                    },
                    "f:metadata": {
                        "f:labels": {
                            "f:kustomize.toolkit.fluxcd.io/name": {},
                            "f:kustomize.toolkit.fluxcd.io/namespace": {}
                        }
                    }
                },
                "manager": "kustomize-controller",
                "operation": "Apply",
                "time": "2025-04-28T23:53:14Z"
            }
        ],
        "name": "cluster-config",
        "namespace": "flux-system",
        "resourceVersion": "834225",
        "uid": "3d7344d0-8960-4c77-acc2-0abc52da29be"
    }
}

Configuring Substitution

We now have an encrypted ConfigMap that we can use to securely store secrets in our cluster. To utilize these secrets we need to configure the post build substitution on any kustomizations that we want to reference these values in. Making sure to apply substitution to every single kustomization would be tedious, but lucky for us, we have structured our flux repo in such a way that makes our lives easier. We can apply ‘patches’ to higher level kustomizations that will apply changes to other kustomizations applied further down the hierarchy. In our case, we can apply patches to the clusters/truek8s/apps.yaml and clusters/truek8s/infrastructure.yaml kustomizations, and all other resources will automatically inherit our substitution configuration. This is one of the reasons it’s so important to consider how you’ll structure your repository.

Go ahead and add the following patch definition to the apps and infrastructure kustomizations. I’ll provide apps as an example, but the content is exactly the same for both (or any other kustomizations you wish to apply it to in your own cluster).

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps/production
  prune: true
  wait: true
  patches:
      - patch: |-
          apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
            name: not-used
          spec:
            decryption:
              provider: sops
              secretRef:
                name: sops-age
            postBuild:
              substituteFrom:
                - kind: ConfigMap
                  name: cluster-config
        target:
          group: kustomize.toolkit.fluxcd.io
          kind: Kustomization
          labelSelector: substitution.flux.home.arpa/disabled notin (true)

It’s that easy! Don’t forget to commit your changes.

git add -A && git commit -m "patched apps and infrastructure"
git push

flux events --watch

Testing Variable Substitution.

We can verify everything is working together by testing out substitution in a helm release. In the last part of the series we deployed podinfo, and that will be the perfect app to test substitution on. Edit the /apps/general/podinfo/podinfo.yaml file, adding a reference to the value we created in our ConfigMap:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo
  namespace: podinfo
spec:
  interval: 10m
  timeout: 5m
  chart:
    spec:
      chart: podinfo
      sourceRef:
        kind: HelmRepository
        name: podinfo
        namespace: flux-system
      interval: 5m
  releaseName: podinfo
  values:
    extraEnvs:
      - name: MULTIPLE_VALUES
        value: ${test}

Commit your changes, and wait for the podinfo helm release to reconcile. Now check the chart values with helm and observe whether or not the test variable was substituted. It should have been replaced with ‘Value’

$ helm get values -n podinfo podinfo
USER-SUPPLIED VALUES:
extraEnvs:
- name: MULTIPLE_VALUES
  value: Value

Sources

• major.io: Encrypted gitops secrets with flux and age
• fluxcd.io: Manage Kubernetes secrets with SOPS
• fluxcd.io: Post build variable substitution
• fluxcd.io: Kustomize questions
• github.org: sops
• github.org: age