Quick Start Instructions

This are the instructions to quickly deploy Kuberentes Pi-cluster using the following tools:

  • cloud-init: to automate initial OS installation/configuration on each node of the cluster
  • Ansible: to automatically configure cluster nodes, install and configure external services (DNS, DHCP, Firewall, S3 Storage server, Hashicorp Vautl) install K3S, and bootstraping cluster through installation and configuration of ArgoCD
  • Argo CD: to automatically deploy Applications to Kuberenetes cluster from manifest files in Git repository.

Preparing the Ansible Control node

  • Set-up a Ubuntu Server VM in your laptop to become ansible control node pimaster and create the SSH public/private keys needed for connecting remotely to the servers

    Follow instructions in “Ansible Control Node”.

  • Clone Pi-Cluster Git repo or download using the ‘Download ZIP’ link on GitHub.

    git clone https://github.com/ricsanfre/pi-cluster.git
    
  • Edit GPG_NAME and GPG_EMAIL variables in Makefile

  • Prepare Ansible execution environment:

    make prepare-ansible
    

    Ansible playbooks depend on external roles that need to be installed, command ansible-galaxy install -r requirements.yml is executed, GPG key for encrypting passwords created, ansible vault automatic encrytpion configured, and pi-cluster credentials file (vault.yml) created and encrypted.

Ansible configuration

Inventory file

Adjust ansible/inventory.yml inventory file to meet your cluster configuration: IPs, hostnames, number of nodes, etc.

Configuring ansible remote access

The UNIX user to be used in remote connection (i.e.: ansible) user and its SSH key file location need to be specified

  • Modify ansible/group_vars/all.yml to set the UNIX user to be used by Ansible in the remote connection (default value ansible)

  • Modify ansible/ansible.cfg file to include the path to the SSH key of the ansible user used in remote connections (private_key_file variable)

    # SSH key
    private_key_file = $HOME/ansible-ssh-key.pem
    

Encrypting secrets/key variables

All secrets/key/passwords variables are stored in a dedicated file, vars/vault.yml, so this file can be encrypted using Ansible Vault

vault.yml file is a Ansible vars file containing just a unique yaml variable, vault: a yaml dictionary containing all keys/passwords used by the different cluster components.

vault.yml sample file is like this:

---
vault:
  # K3s secrets
  k3s:
    k3s_token: s1cret0
  # traefik secrets
  traefik:
    basic_auth:
      user: admin
      passwd: s1cret0
  # Minio S3 secrets
  minio:
    root:
      user: root
      key: supers1cret0
    restic:
      user: restic
      key: supers1cret0
....

The manual steps to encrypt passwords/keys used in all Playbooks is the following:

  1. Edit content var/vault.yml file specifying your own values for each of the key/password/secret specified.

  2. Encrypt file using ansible-vault

    ansible-vault encrypt vault.yml
    

    The command ask for a ansible vault password to encrypt the file. After executing the command the file vault.yml is encrypted. Yaml content file is not readable.

Automate Ansible Vault decryption with GPG

When using encrypted vault.yaml file all playbooks executed with ansible-playbook command need the argument --ask-vault-pass, so the password used to encrypt vault file can be provided when starting the playbook.

ansible-playbook playbook.yml --ask-vault-pass

Ansible vault password decryption can be automated using --vault-password-file parameter , instead of manually providing the password with each execution (--ask-vault-pass).

Ansible vault password file can contain the password in plain-text or a script able to obtain the password.

vault-password-file location can be added to ansible.cfg file, so it is not needed to pass as parameter each time ansible-playbook command is executed

Linux GPG will be used to encrypt Ansible Vault passphrase and automatically obtain the vault password using a vault-password-file script.

  • GnuPG Installation and configuration

    In Linux GPG encryption can be used to encrypt/decrypt passwords and tokens data using a GPG key-pair

    GnuPG package has to be installed and a GPG key pair need to be created for encrytion/decryption

    • Step 1. Install GnuPG packet

      sudo apt install gnupg 
      

      Check if it is installed

      gpg --help
      
    • Step 2. Generating Your GPG Key Pair

      GPG key-pair consist on a public and private key used for encrypt/decrypt

      gpg --gen-key
      

      The process requires to provide a name, email-address and user-id which identify the recipient

      The output of the command is like this:

        gpg (GnuPG) 2.2.4; Copyright (C) 2017 Free Software Foundation, Inc.
        This is free software: you are free to change and redistribute it.
        There is NO WARRANTY, to the extent permitted by law.
      
        Note: Use "gpg --full-generate-key" for a full featured key generation dialog.
      
        GnuPG needs to construct a user ID to identify your key.
      
        Real name: Ricardo
        Email address: ricsanfre@gmail.com
        You selected this USER-ID:
            "Ricardo <ricsanfre@gmail.com>"
      
        Change (N)ame, (E)mail, or (O)kay/(Q)uit? O
        We need to generate a lot of random bytes. It is a good idea to perform
        some other action (type on the keyboard, move the mouse, utilize the
        disks) during the prime generation; this gives the random number
        generator a better chance to gain enough entropy.
        We need to generate a lot of random bytes. It is a good idea to perform
        some other action (type on the keyboard, move the mouse, utilize the
        disks) during the prime generation; this gives the random number
        generator a better chance to gain enough entropy.
        gpg: /home/ansible/.gnupg/trustdb.gpg: trustdb created
        gpg: key D59E854B5DD93199 marked as ultimately trusted
        gpg: directory '/home/ansible/.gnupg/openpgp-revocs.d' created
        gpg: revocation certificate stored as '/home/ansible/.gnupg/openpgp-revocs.d/A4745167B84C8C9A227DC898D59E854B5DD93199.rev'
        public and secret key created and signed.
      
        pub   rsa3072 2021-08-13 [SC] [expires: 2023-08-13]
              A4745167B84C8C9A227DC898D59E854B5DD93199
        uid                      Ricardo <ricsanfre@gmail.com>
        sub   rsa3072 2021-08-13 [E] [expires: 2023-08-13]
      
      

      During the generation process you will be prompted to provide a passphrase.

      This passphrase is needed to decryp

  • Generate Vault password and store it in GPG

    Generate the password to be used in ansible-vault encrypt/decrypt process and ecrypt it in using GPG

    • Step 1. Install pwgen packet

        sudo apt install pwgen 
      
    • Step 2: Generate Vault password and encrypt it using GPG. Store the result as a file in $HOME/.vault

      mkdir -p $HOME/.vault
      pwgen -n 71 -C | head -n1 | gpg --armor --recipient <recipient> -e -o $HOME/.vault/vault_passphrase.gpg
      

      where <recipient> must be the email address configured during GPG key creation.

    • Step 3: Generate a script vault_pass.sh

      #!/bin/sh
      gpg --batch --use-agent --decrypt $HOME/.vault/vault_passphrase.gpg
      
    • Step 4: Modify ansible.cfg file, so you can omit the --vault-password-file argument.

      [defaults]
      vault_password_file=vault_pass.sh
      
  • Encrypt vautl.yaml file using ansible-vault and GPG password

    ansible-vault encrypt vault.yaml
    

    This time only your GPG key passphrase will be asked to automatically encrypt/decrypt the file

Vault credentials generation

Execute playbook to generate ansible vault variable file (var/vault.yml) containing all credentials/passwords. Random generated passwords will be generated for all cluster services.

Execute the following command:

ansible-playbook create_vault_credentials.yml

Credentials for external cloud services (IONOS DNS API credentials) will be asked during the execution of the script.

Modify Ansible Playbook variables

Adjust ansible playbooks/roles variables defined within group_vars, host_vars and vars directories to meet your specific configuration.

The following table shows the variable files defined at ansible’s group and host levels

Group/Host Variable file Nodes affected
ansible/group_vars/all.yml all nodes of cluster + gateway node + pimaster
ansible/group_vars/control.yml control group: gateway node + pimaster
ansible/group_vars/k3s_cluster.yml all nodes of the k3s cluster
ansible/group_vars/k3s_master.yml K3s master nodes
ansible/host_vars/gateway.yml gateway node specific variables

The following table shows the variable files used for configuring the storage, backup server and K3S cluster and services.

Specific Variable File Configuration
ansible/vars/picluster.yml K3S cluster and external services configuration variables
ansible/vars/dedicated_disks/local_storage.yml Configuration nodes local storage: Dedicated disks setup
ansible/vars/centralized_san/centralized_san_target.yml Configuration iSCSI target local storage and LUNs: Centralized SAN setup
ansible/vars/centralized_san/centralized_san_initiator.yml Configuration iSCSI Initiator: Centralized SAN setup

Installing the nodes

Update Raspberry Pi firmware

Update firmware in all Raspberry-PIs following the procedure described in “Raspberry PI firmware update”

Install gateway node

Install gateway Operating System on Rapberry PI.

The installation procedure followed is the described in “Ubuntu OS Installation” using cloud-init configuration files (user-data and network-config) for gateway, depending on the storage setup selected:

Storage Configuration User data Network configuration
Dedicated Disks user-data network-config
Centralized SAN user-data network-config

Configure gateway node

For automatically execute basic OS setup tasks and configuration of gateway’s services (DNS, DHCP, NTP, Firewall, etc.), execute the command:

make gateway-setup

Install cluster nodes.

Once gateway is up and running the rest of the nodes can be installed and connected to the LAN switch, so they can obtain automatic network configuration via DHCP.

Install node1-5 Operating System on Raspberry Pi

Follow the installation procedure indicated in “Ubuntu OS Installation” using the corresponding cloud-init configuration files (user-data and network-config) depending on the storage setup selected. Since DHCP is used there is no need to change default /boot/network-config file located in the ubuntu image.

Storage Architeture node1 node2 node3 node4 node5
Dedicated Disks user-data user-data user-data user-data user-data
Centralized SAN user-data user-data user-data user-data user-data

Configure cluster nodes

For automatically execute basic OS setup tasks (DNS, DHCP, NTP, etc.), execute the command:

make nodes-setup

Configuring external services (Minio and Hashicorp Vault)

Install and configure S3 Storage server (Minio), and Secret Manager (Hashicorp Vault) running the command:

make external-services

Ansible Playbook assumes S3 server is installed in node1 and Hashicorp Vault in gateway.

Configuring OS level backup (restic)

Automate backup tasks at OS level with restic in all nodes (node1-node5 and gateway) running the command:

make configure-os-backup

Minio S3 server running in node1 will be used as backup backend.

Kubernetes Applications (GitOps)

ArgoCD is used to deploy automatically packaged applications contained in the repository. These applications are located in /argocd directory.

  • Modify Root application (App of Apps pattern) to point to your own repository

    Edit file /argocd/bootstrap/root/values.yaml.

    gitops.repo should point to your own cloned repository.

    gitops:
      repo: https://github.com/<your-user>/pi-cluster 
    
  • Tune parameters of the different packaged Applications to meet your specific configuration

    Edit values.yaml file of the different applications located in /argocd/system directory.

K3S

K3S Installation

To install K3S cluster, execute the command:

make k3s-install

K3S Bootstrap

To bootstrap the cluster, run the command:

make k3s-bootstrap

Argo CD will be installed and it will automatically deploy all cluster applications automatically from git repo

  • argocd\bootstrap\root: Containing root application (App of Apss ArgoCD pattern)
  • argocd\system\<app>: Containing manifest files for application

K3s Cluster reset

If you mess anything up in your Kubernetes cluster, and want to start fresh, the K3s Ansible playbook includes a reset playbook, that you can use to remove the installation of K3S:

make k3s-reset

Shutting down the Raspberry Pi Cluster

To automatically shut down the Raspberry PI cluster, Ansible can be used.

Kubernetes graceful node shutdown feature is enabled in the culster. This feature is documented here. and it ensures that pods follow the normal pod termination process during the node shutdown.

For doing a controlled shutdown of the cluster execute the following commands

  • Step 1: Shutdown K3S workers nodes:

    make shutdown-k3s-worker
    
  • Step 2: Shutdown K3S master nodes:

    make shutdown-k3s-master
    
  • Step 3: Shutdown gateway node:

    make shutdown-gateway
    

shutdown commands connects to each Raspberry PI in the cluster and execute the command sudo shutdown -h 1m, commanding the raspberry-pi to shutdown in 1 minute.

After a few minutes, all raspberry pi will be shutdown. You can notice that when the Switch ethernet ports LEDs are off. Then it is safe to unplug the Raspberry PIs.

Updating Ubuntu packages

To automatically update Ubuntu OS packages, run the following command:

make os-upgrade

This playbook automatically updates OS packages to the latest stable version and it performs a system reboot if needed.


Last Update: Jan 23, 2023

Comments: