vault-secrets-operator

The Vault Secrets Operator creates Kubernetes secrets from Vault. The idea behind the Vault Secrets Operator is to manage secrets in Kubernetes cluster using a secure GitOps based workflow. With the help of the Vault Secrets Operator you can commit your secrets to your git repository using a custom resource. If you apply these secrets to your Kubernetes cluster the Operator will lookup the real secret in Vault and creates the corresponding Kubernetes secret. If you are using something like Sealed Secrets for this workflow the Vault Secrets Operator can be used as replacement for this.

Installation

The Vault Secrets Operator can be installed via Helm. A list of all configurable values can be found here. The chart assumes a vault server running at http://vault:8200, but can be overidden by specifying --set vault.address=https://vault.example.com

helm upgrade --install vault-secrets-operator oci://ghcr.io/ricoberger/charts/vault-secrets-operator --version 

Prepare Vault

The Vault Secrets Operator supports the KV Secrets Engine - Version 1 and KV Secrets Engine - Version 2. To create a new secret engine under a path named kvv1 and kvv2, you can run the following command:

vault secrets enable -path=kvv1 -version=1 kv
vault secrets enable -path=kvv2 -version=2 kv

After you have enabled the secret engine, create a new policy for the Vault Secrets Operator. The operator only needs read access to the paths you want to use for your secrets. To create a new policy with the name vault-secrets-operator and read access to the kvv1 and kvv2 path, you can run the following command:

cat  password= policies=

Use the following commands to set the environment variables for the activation of the UserPass auth method:

export VAULT_AUTH_METHOD=userpass
export VAULT_USER=
export VAULT_PASSWORD=
export VAULT_TOKEN_MAX_TTL=120

When you deploy the Vault Secrets Operator via Helm chart you have to set the vault.authMethod property to userpass in the values.yaml file, to use the UserPass auth method instead of the default Token auth method.

vault:
  authMethod: userpass

AWS Auth Method

You can use either ec2 or iam auth types on eks clusters to authenticate against the Vault API. here Then you can enable the auth method with the following environment variables:

export VAULT_AUTH_METHOD=aws
export VAULT_AWS_PATH=auth/aws
export VAULT_AWS_ROLE=vault-secrets-operator
export VAULT_AWS_AUTH_TYPE=iam

If you deploy the Vault Secrets Operator via Helm you have to set the vault.authMethod, vault.awsPath, vault.awsRole and vault.awsAuthType values in the values.yaml file.

Azure Auth Method

You can use the managed system identity provided on aks cluster to authenticate against the Vault API, to do that you will need to setup an auth backend as described here Then you can setup the auth method with the following environment variables:

export VAULT_AUTH_METHOD=azure
export VAULT_AZURE_PATH=auth/azure
export VAULT_AZURE_ROLE=default
export VAULT_AZURE_ISSCALESET=true # Set this to true if the kubernetes nodes are in a vmss and not isolated vm (default in aks)

If you deploy the Vault Secrets Operator via Helm you have to set the vault.authMethod, vault.azurepath, vault.azureRole, vault.azureScaleset values in the values.yaml file.

GCP Auth Method

You can use either gce or iam auth types on gke clusters to authenticate against the Vault API. here Then you can enable the auth method with the following environment variables:

export VAULT_AUTH_METHOD=gcp
export VAULT_GCP_PATH=auth/gcp
export VAULT_GCP_ROLE=vault-secrets-operator
export VAULT_GCP_AUTH_TYPE=iam

If you deploy the Vault Secrets Operator via Helm you have to set the vault.authMethod, vault.gcpPath, vault.gcpRole and vault.gcpAuthType values in the values.yaml file.

Usage

Secret Engine

Create two Vault secrets example-vaultsecret:

vault kv put kvv1/example-vaultsecret foo=bar hello=world

vault kv put kvv2/example-vaultsecret foo=bar
vault kv put kvv2/example-vaultsecret hello=world
vault kv put kvv2/example-vaultsecret foo=bar hello=world

Deploy the custom resource kvv1-example-vaultsecret to your Kubernetes cluster:

apiVersion: ricoberger.de/v1alpha1
kind: VaultSecret
metadata:
  name: kvv1-example-vaultsecret
spec:
  keys:
    - foo
  path: kvv1/example-vaultsecret
  type: Opaque

The Vault Secrets Operator creates a Kubernetes secret named kvv1-example-vaultsecret with the type Opaque from this CR:

apiVersion: v1
data:
  foo: YmFy
kind: Secret
metadata:
  labels:
    created-by: vault-secrets-operator
  name: kvv1-example-vaultsecret
type: Opaque

You can also omit the keys spec to create a Kubernetes secret which contains all keys from the Vault secret:

apiVersion: v1
data:
  foo: YmFy
  hello: d29ybGQ=
kind: Secret
metadata:
  labels:
    created-by: vault-secrets-operator
  name: kvv1-example-vaultsecret
type: Opaque

To deploy a custom resource kvv2-example-vaultsecret, which uses the secret from the KV Secrets Engine - Version 2 you can use the following:

apiVersion: ricoberger.de/v1alpha1
kind: VaultSecret
metadata:
  name: kvv2-example-vaultsecret
spec:
  path: kvv2/example-vaultsecret
  type: Opaque

The Vault Secrets Operator will create a secret which looks like the following:

apiVersion: v1
data:
  foo: YmFy
  hello: d29ybGQ=
kind: Secret
metadata:
  labels:
    created-by: vault-secrets-operator
  name: kvv2-example-vaultsecret
type: Opaque

For secrets using the KVv2 secret engine you can also specify the version of the secret you want to deploy:

apiVersion: ricoberger.de/v1alpha1
kind: VaultSecret
metadata:
  name: kvv2-example-vaultsecret
spec:
  path: kvv2/example-vaultsecret
  type: Opaque
  version: 2

The resulting Kubernetes secret will be:

apiVersion: v1
data:
  hello: d29ybGQ=
kind: Secret
metadata:
  labels:
    created-by: vault-secrets-operator
  name: kvv2-example-vaultsecret
type: Opaque

The spec.type and spec.keys fields are handled in the same way for both versions of the KV secret engine. The spec.version field is only processed, when the secret is saved under a KVv2 secret engine. If you specified the VAULT_RECONCILIATION_TIME environment variable with a value greater than 0 every secret is reconciled after the given time (in seconds). This means, when you do not specify spec.version, the Kubernetes secret will be automatically updated if the Vault secret changes. To set the VAULT_RECONCILIATION_TIME environment variable in the Helm chart the vault.reconciliationTime value can be used.

The binary data stored in vault requires base64 encoding. the spec.isBinary can be used to prevent such data get base64 encoded again when store as secret in k8s.

For example, let's set foo to the bar in base64 encoded format (i.e. YmFyCg==).

vault kv put kvv1/example-vaultsecret foo=YmFyCg==

You can specify spec.isBinary to indicate this is a binary data which is already in base64 encoded format:

apiVersion: ricoberger.de/v1alpha1
kind: VaultSecret
metadata:
  name: kvv1-example-vaultsecret
spec:
  keys:
    - foo
  isBinary: true
  path: kvv1/example-vaultsecret
  type: Opaque

The resulting Kubernetes secret will be:

apiVersion: v1
data:
  foo: YmFyCg==
kind: Secret
metadata:
  labels:
    created-by: vault-secrets-operator
  name: kvv1-example-vaultsecret
type: Opaque

The value for foo stays as YmFyCg== which does not get base64 encoded again.

It is also possible to change the default reconciliation strategy from Replace to Merge via the reconcileStrategy key in the CRD. For the default Replace strategy the complete secret is replaced. If you have an existing secret you can choose the Merge strategy to add the keys from Vault to the existing secret.

Creating a secret from multiple Vault paths

A single Kubernetes secret can be created from multiple Vault secrets by using the paths property. All secrets referenced by path and paths are merged into a single Kubernetes secret. The secrets share the same top-level options (e.g. keys, version, isBinary, secretEngine and vaultNamespace) and multiple paths are only supported for the kv secret engine.

The paths are processed in order: the optional path field is used first, followed by the entries in paths. If multiple Vault secrets contain the same key, the value from the first path which provides that key is used.

apiVersion: ricoberger.de/v1alpha1
kind: VaultSecret
metadata:
  name: prometheus-scrape-configs
  namespace: monitoring
spec:
  keys:
    - kuma_password
  path: kvv2/ORG/secret1
  paths:
    - kvv2/ORG/secret2
  type: Opaque

When the templates property is used, all secrets are additionally available per path via the .SecretsPaths context, so that duplicate keys from different paths can still be accessed. See Templating context for details.

Using templated secrets

When straight-forward secrets are not sufficient, and the target secrets need to be formatted in a certain way, you can use basic templating to format the secrets. There are multiple uses for this:

• Generate URIs which contain secrets
• Format secrets in a specific way, for example when using the Helm Operator which can use secrets as a source for its Helm chart parameterisation, but they have to be in YAML format wrapped inside a secret, like secretGenerator from Kustomize also generates.

To do this, specify keys under spec.templates, containing a valid template string. When templates is defined, the standard generation of secrets is disabled, and only the defined templates will be generated.

The templating uses the standard Go templating engine, also used in tools such as Helm or Gomplate. The main differentiator here is that the {% and %} delimiters are used to prevent conflicts with standard Go templating tools such as Helm, which use {{ and }} for this.

The available functions during templating are the set offered by the Sprig library (similar to Helm, but different from Gomplate), excluding the following functions for security-reasons or their non-idempotent nature to avoid reconciliation problems:

• genPrivateKey
• genCA
• genSelfSignedCert
• genSignedCert
• htpasswd
• getHostByName
• Random functions
• Date/time functionality
• Environment variable functions (for security reasons)

Templating context

The context available in the templating engine contains the following items:

• .Secrets: Map with all the secrets fetched from vault. Key = secret name, Value = secret value. When multiple Vault paths are configured (see Creating a secret from multiple Vault paths) and they contain the same key, the value from the first path wins.
• .SecretsPaths: Ordered list with the secrets of each configured Vault path, so that all secrets are available even when multiple paths contain the same key. Each entry has a .Path (