Secret Management (Vault)

HashiCorp Vault is used as Secret Management solution for Raspberry PI cluster. All cluster secrets (users, passwords, api tokens, etc) will be securely encrypted and stored in Vault.

Vault will be deployed as a external service, not running as a Kuberentes service, so it can be used by GitOps solution, ArgoCD, to deploy automatically all cluster services.

Vault could be installed as Kuberentes service, deploying it using an official Helm Chart or a community operator like Banzai Bank-Vault.

Installing Vault as Kubernetes service will drive us to a chicken/egg situation if we want to use Vault as only source of secrets/credentials for all Kuberentes services deployed. Vault requires to have Block storage solution (Longhorn) deployed first since its POD needs Perstistent Volumes, and to install Longhorn, a few secrets need to be provided to configure its backup (Minio credentials).

External Secrets Operator will be used to automatically generate the Kubernetes Secrets from Vault data that is needed to deploy the different services using ArgoCD.


Vault installation

Vault installation and configuration tasks have been automated with Ansible developing a role: ricsanfre.vault. This role, installs Vault Server, initialize it and install a systemd service to automatically unseal it whenever vault server is restarted.

Vault installation from binaries

Instead of installing Vault using official Ubuntu packages, installation will be done manually from binaries, so the version to be installed can be decided.

  • Step 1. Create vault’s UNIX user/group

    vault user is a system user, not login allowed

    sudo groupadd vault 
    sudo useradd vault -g vault -r -s /sbin/nologin
  • Step 2. Create vault’s storage directory

    sudo mkdir /var/lib/vault
    chown -R vault:vault /var/lib/vault
    chmod -R 750 /vault/lib/vault
  • Step 3. Create vault’s config directories

    sudo mkdir -p /etc/vault
    sudo mkdir -p /etc/vault/tls
    sudo mkdir -p /etc/vault/policy
    sudo mkidr -p /etc/vault/plugin
    chown -R vault:vault /etc/vault
    chmod -R 750 /etc/vault
  • Step 4: Create vault’s log directory

    sudo mkdir /var/log/vault
    chown -R vault:vault /var/log/vault
    chmod -R 750 /vault/log/vault
  • Step 5. Download server binary (vault) and copy them to /usr/local/bin

     unzip vault_<version>_linux_<arch>.zip
     chmod +x vault
     sudo mv vault /usr/local/bin/.

    where <arch> is amd64 or arm64, and <version> is vault version (for example: 1.12.2).

  • Step 6. Create Vault TLS certificate

    In case you have your own domain, a valid TLS certificate signed by Letsencrypt can be obtained for Minio server, using Certbot.

    See certbot installation instructions in CertManager - Letsencrypt Certificates Section. Those instructions indicate how to install certbot using DNS challenge with IONOS DNS provider (my DNS provider). Similar procedures can be followed for other DNS providers.

    Letsencrypt using HTTP challenge is avoided for security reasons (cluster services are not exposed to public internet).

    If generating valid TLS certificate is not possible, selfsigned certificates with a custom CA can be used instead.

    Follow this procedure for creating a self-signed certificate for Vault Server

    1. Create a self-signed CA key and self-signed certificate

      openssl req -x509 \
             -sha256 \
             -nodes \
             -newkey rsa:4096 \
             -subj "/CN=Ricsanfre CA" \
             -keyout rootCA.key -out rootCA.crt
    1. Create a TLS certificate for Vault server signed using the custom CA

      openssl req -new -nodes -newkey rsa:4096 \
                  -keyout vault.key \
                  -out vault.csr \
                  -batch \
                  -subj "/C=ES/ST=Madrid/L=Madrid/O=Ricsanfre CA/OU=picluster/"
       openssl x509 -req -days 365000 -set_serial 01 \
             -extfile <(printf "") \
             -in vault.csr \
             -out vault.crt \
             -CA rootCA.crt \
             -CAkey rootCA.key

    Once the certificate is created, public certificate and private key need to be installed in Vault server following this procedure:

    1. Copy public certificate vault.crt as /etc/vault/tls/vault.crt

      sudo cp vault.crt /etc/vault/tls/public.crt
      sudo chown vault:vault /etc/vault/tls/public.crt
    2. Copy private key vault.key as /etc/vault/tls/vault.key

      cp vault.key /etc/vault/tls/vault.key
      sudo chown vault:vault /etc/vault/tls/vault.key
    3. Copy CA certificate rootCA.crt as /etc/vault/tls/vault-ca.crt

      cp rootCA.crt /etc/vault/tls/vault-ca.crt
      sudo chown vault:vault /etc/vault/tls/vault-ca.crt
  • Step 7: Create vault config file /etc/vault/vault_main.hcl

    cluster_addr  = "https://<node_ip>:8201"
    api_addr      = "https://<node_ip>:8200"
    plugin_directory = "/etc/vault/plugin"
    disable_mlock = true
    listener "tcp" {
      address     = ""
      tls_cert_file      = "/etc/vault/tl/vault.crt"
      tls_key_file       = "/etc/vault/tls/vault.key"
      tls_disable_client_certs = true
    storage "raft" {
      path    = /var/lib/vault

    Vault is configured, as a single node of HA cluster, with the following parameters:

    • Node’s URL address to be used in internal communications between nodes of the cluster. (cluster_addr and api_addr)
    • Vault server API listening in all node’s addresses at port 8200: (listener "tcp" address=
    • TLS certifificates are stored in /etc/vault/tls
    • Client TLS certificates validation is disabled (tls_disable_client_certs)
    • Vault is configured to use integrated storage Raft data dir /var/lib/vault
    • Disables the server from executing the mlock syscall (disable_mlock) recommended when using Raft storage
  • Step 8. Create systemd vault service file /etc/systemd/system/vault.service

    Description="HashiCorp Vault - A tool for managing secrets"
    AmbientCapabilities=CAP_SYSLOG CAP_IPC_LOCK
    CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
    ExecStart=/bin/sh -c 'exec /vault server -config=/etc/vault/vault_main.hcl -log-level=info'
    ExecReload=/bin/kill --signal HUP $MAINPID

    This service start vault server using vault UNIX group and executing the following startup command:

    /usr/local/vault server -config=/etc/vault/vault_main.hcl -log-level=info
  • Step 9. Enable vault systemd service and start it

    sudo systemctl enable vault.service
    sudo systemctl start vault.service
  • Step 10. Check vault server status

    export VAULT_ADDR=https://<vault_ip>:8200
    export VAULT_CACERT=/etc/vault/tls/vault-ca.crt
    vault status

    The output should be like the following

    Key                Value
    ---                -----
    Seal Type          shamir
    Initialized        false
    Sealed             true
    Total Shares       0
    Threshold          0
    Unseal Progress    0/0
    Unseal Nonce       n/a
    Version            1.12.2
    Build Date         2022-11-23T12:53:46Z
    Storage Type       raft
    HA Enabled         true

    It shows Vault server status as not initialized (Initialized = false) and sealed (Sealed = true).

Vault initialization and useal

During initialization, Vault generates a root key, which is stored in the storage backend alongside all other Vault data. The root key itself is encrypted and requires an unseal key to decrypt it.

Unseal process, where uneal keys are provided to rebuid the root key, need to be completed every time vault server is started.

The default Vault configuration uses Shamir’s Secret Sharing to split the root key into a configured number of shards (referred as key shares, or unseal keys). A certain threshold of shards is required to reconstruct the root key, which is then used to decrypt the Vault’s encryption key.

To initialize vault vault operator init command must be used.

vault operator init -key-shares=1 -key-threshold=1 -format=json > /etc/vault/unseal.json

where number of key shares (-key-shares) and threshold (-key-threshold) is set to 1. Only one key is needed to unseal vault.

The vault init command output is redirected to a file (/etc/vautl/unseal.json) containing unseal keys values and root token needed to connect to vault.

  "unseal_keys_b64": [
  "unseal_keys_hex": [
  "unseal_shares": 1,
  "unseal_threshold": 1,
  "recovery_keys_b64": [],
  "recovery_keys_hex": [],
  "recovery_keys_shares": 0,
  "recovery_keys_threshold": 0,
  "root_token": "hvs.AJxt0CgXT9BcVe5dMNeI0Unm"

vault status shows Vault server initialized but sealed

vault status
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       1
Threshold          1
Unseal Progress    0/1
Unseal Nonce       n/a
Version            1.12.2
Build Date         2022-11-23T12:53:46Z
Storage Type       raft
HA Enabled         true

To unseal vault vault operator unseal command need to be executed, providing unseal keys generated during initialization process.

Using the key stored in unseal.json file the following command can be executed:

vault operator unseal $(jq -r '.unseal_keys_b64[0]' /etc/vault/unseal.json)

vault status shows Vault server initialized and unsealed

vault status
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       1
Threshold          1
Unseal Progress    0/1
Unseal Nonce       n/a
Version            1.12.2
Build Date         2022-11-23T12:53:46Z
Storage Type       raft
HA Enabled         true

Vault automatic unseal

A systemd service can be created to automatically unseal vault every time it is started.

  • Step 1: Create a script (/etc/vault/ for automating the unseal process using the keys stored in /etc/vault/unseal.json

    #!/usr/bin/env sh
    #Define a timestamp function
    timestamp() {
    date "+%b %d %Y %T %Z"
    if [ true = "$SKIP_TLS_VERIFY" ]
    # Add timestamp
    echo "$(timestamp): Vault-useal started" | tee -a $LOG
    echo "-------------------------------------------------------------------------------" | tee -a $LOG
    initialized=$(curl $CURL_PARAMS $URL/v1/sys/health | jq '.initialized')
    if [ true = "$initialized" ]
      echo "$(timestamp): Vault already initialized" | tee -a $LOG
      while true
        status=$(curl $CURL_PARAMS $URL/v1/sys/health | jq '.sealed')
        if [ true = "$status" ]
            echo "$(timestamp): Vault Sealed. Trying to unseal" | tee -a $LOG
            # Get keys from json file
            for i in `jq -r '.keys[]' $KEYS_FILE` 
              do curl $CURL_PARAMS --request PUT --data "{\"key\": \"$i\"}" $URL/v1/sys/unseal
        sleep 10
            echo "$(timestamp): Vault unsealed" | tee -a $LOG
      echo "$(timestamp): Vault not initialized yet"
  • Step 2: Create systemd vault service file /etc/systemd/system/vault-unseal.service

    Description=Vault Unseal
    ExecStartPre=/bin/sleep 10
    ExecStart=/bin/sh -c '/etc/vault/'
    [Install] vault.service

    This service is defined as part of vault.service (PartOf), so stopping/starting vault.service is propagated to this service.

  • Step 3. Enable vault systemd service and start it

    sudo systemctl enable vault-unseal.service
    sudo systemctl start vault-unseal.service

Vault configuration

Once vault is unsealed following configuration requires to provide vault’s root token generated during initialization procces. See root_token in unseal.json output.

export VAULT_TOKEN=$(jq -r '.root_token' /etc/vault/unseal.json)

Enabling KV secrets

Enable KV (KeyValue) secrets engine to manage static secrets.

vault secrets enable -version=2 -path=secret kv

This command enables KV version 2 at path /secret

Vault policies

Create vault policies to read and read/write KV secrets

  • Read-write policy

    Create file /etc/vault/policy/secrets-write.hcl

    path "secret/*" {
      capabilities = [ "create", "read", "update", "delete", "list", "patch" ]

    Add policy to vault

    vault policy write readwrite /etc/vault/policy/secrets-readwrite.hcl
  • Read-only policy

    Create file /etc/vault/policy/secrets-read.hcl

    path "secret/*" {
      capabilities = [ "read" ]

    Add policy to vault

    vault policy write readonly /etc/vault/policy/secrets-read.hcl

Testing policies:

  • Generate tokens for read and write policies

    READ_TOKEN=$(vault token create -policy="readonly" -field=token)
    WRITE_TOKEN=$(vault token create -policy="readwrite" -field=token)
  • Try write a secret using read token

    vault kv put secret/secret1 user="user1" password="s1cret0"

    Permission denied error:

    Code: 403. Errors:
    * 1 error occurred:
      * permission denied
  • Try write a secret using write token

    vault kv put secret/secret1 user="user1" password="s1cret0"

    The secret is stored with success:

    === Secret Path ===
    ======= Metadata =======
    Key                Value
    ---                -----
    created_time       2023-01-02T11:04:21.01853116Z
    custom_metadata    <nil>
    deletion_time      n/a
    destroyed          false
    version            1
  • Secret can be read using both tokens

    vault kv get secret/secret1
    === Secret Path ===
    ======= Metadata =======
    Key                Value
    ---                -----
    created_time       2023-01-02T11:04:21.01853116Z
    custom_metadata    <nil>
    deletion_time      n/a
    destroyed          false
    version            1
    ====== Data ======
    Key         Value
    ---         -----
    password    s1cret0
    user        user1

Kubernetes Auth Method

Enabling Vault kubernetes auth method to authenticate with Vault using a Kubernetes Service Account Token. This method of authentication makes it easy to introduce a Vault token into a Kubernetes Pod.

  • Step 1. Create vault namespace

    kubectl create namespace vault
  • Step 2. Create service account vault-auth to be used by Vault kuberentes authentication

    apiVersion: v1
    kind: ServiceAccount
      name: vault-auth
      namespace: vault
  • Step 3. Add proper permissions to service account

    Vault kubernetes authentication method accesses the Kubernetes TokenReview API to validate the provided JWT is still valid. Service Accounts used in this auth method will need to have access to the TokenReview API. If Kubernetes is configured to use RBAC roles, the Service Account should be granted permissions to access this API. Check more details in Vault - Kubernetes Auth Method

    kind: ClusterRoleBinding
      name: role-tokenreview-binding
      namespace: vault
      kind: ClusterRole
      name: system:auth-delegator
      - kind: ServiceAccount
        name: vault-auth
        namespace: vault
  • Step 4. Create long-lived token for vault-auth service account. From Kubernetes v1.24, secrets contained long-lived tokens associated to service accounts are not longer created. See how to create it in Kubernetes documentation

    apiVersion: v1
    kind: Secret
      name: vault-auth-secret
      namespace: vault
      annotations: vault-auth
  • Step 5. Get Service Account token

    KUBERNETES_SA_SECRET_NAME=$(kubectl get secrets --output=json -n vault | jq -r '.items[].metadata | select(.name|startswith("vault-auth")).name')
    TOKEN_REVIEW_JWT=$(kubectl get secret $KUBERNETES_SA_SECRET_NAME -n vault -o jsonpath='{.data.token}' | base64 --decode)
  • Step 6. Get Kubernetes CA cert and API URL

    # Get Kubernetes CA
    kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode > k3s_ca.crt
    # Get Kubernetes Url
    KUBERNETES_HOST=$(kubectl config view -o jsonpath='{.clusters[].cluster.server}')
  • Step 7. Enable Kubernetes auth method

    vault auth enable kubernetes

    Or using Vault API

    curl -k --header "X-Vault-Token:$VAULT_TOKEN" --request POST\
      --data '{"type":"kubernetes","description":"kubernetes auth"}' \
  • Step 8. Configure Vault kubernetes auth method

    vault write auth/kubernetes/config  \
      token_reviewer_jwt="${TOKEN_REVIEW_JWT}" \
      kubernetes_host="${KUBERNETES_HOST}" \

    Or using Vault API:

    KUBERNETES_CA_CERT=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode | awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}')
    curl --cacert /etc/vault/tls/vault_ca.pem --header "X-Vault-Token:$VAULT_TOKEN" --request POST --data '{"kubernetes_host": "'"$KUBERNETES_HOST"'", "kubernetes_ca_cert":"'"$KUBERNETES_CA_CERT"'", "token_reviewer_jwt":"'"$TOKEN_REVIEW_JWT"'"}'

External Secrets Operator installation

External Secrets Operator is installed through its helm chart.

  • Step 1: Add External sercrets repository:
      helm repo add external-secrets
  • Step 2: Fetch the latest charts from the repository:
    helm repo update
  • Step 3: Create namespace
    kubectl create namespace external-secrets
  • Step 4: Install helm chart
    helm install external-secrets \
       external-secrets/external-secrets \
        -n external-secrets \
        --set installCRDs=true
  • Step 5: Create external secrets vault role. Applying read policy

    vault write auth/kubernetes/role/external-secrets \
      bound_service_account_names=external-secrets \
      bound_service_account_namespaces=external-secrets \
      policies=readonly \

    Or using the Vault API

    curl -k --header "X-Vault-Token:$VAULT_TOKEN" --request POST \
      --data '{ "bound_service_account_names": "external-secrets", "bound_service_account_namespaces": "external-secrets", "policies": ["readonly"], "ttl" : "24h"}' \
  • Step 6: Create Cluster Secret Store

     kind: ClusterSecretStore
       name: vault-backend
       namespace: external-secrets
           server: ""
           # caBundle needed if vault TLS is signed using a custom CA.
           # If Vault TLS is valid signed by Letsencrypt this is not needed?
           # ca cert base64 encoded and remobed '\n' characteres"
           # <vault-ca> =`cat vault-ca.pem | base64 | tr -d "\n"`
           caBundle: <vault-ca>
           path: "secret"
           version: "v2"
               mountPath: "kubernetes"
               role: "external-secrets"

    Check ClusterSecretStore status

    kubectl get clustersecretstore -n external-secrets
    vault-backend   10m   Valid    ReadWrite      True
  • Step 7: Create External secret

    kind: ExternalSecret
     name: vault-example
       name: vault-backend
       kind: ClusterSecretStore
       name: mysecret
     - secretKey: password
         key: secret1
         property: password
     - secretKey: user
         key: secret1
         property: user

    Check ExternalSecret status

    kubectl get externalsecret
    NAME            STORE           REFRESH INTERVAL   STATUS         READY
    vault-example   vault-backend   1h                 SecretSynced   True

    Check Secret created

    kubectl get secret mysecret -o yaml


Last Update: Dec 27, 2022