Pi Cluster Documentation

Quick Start Instructions

This are the instructions to quickly deploy Kuberentes Pi-cluster using cloud-init and Ansible Playbooks

Preparing the Ansible Control node

  • Set-up a Ubuntu Server VM in your laptop to become ansible control node pimaster and create the SSH public/private keys needed for connecting remotely to the servers

    Follow instructions in “Ansible Control Node”.

  • Clone Pi-Cluster Git repo or download using the ‘Download ZIP’ link on GitHub.

    git clone https://github.com/ricsanfre/pi-cluster.git

  • Install Ansible requirements:

    Developed Ansible playbooks depend on external roles that need to be installed.

    ansible-galaxy install -r requirements.yml

Ansible playbooks configuration

Inventory file

Adjust inventory.yml inventory file to meet your cluster configuration: IPs, hostnames, number of nodes, etc.

Configuring ansible remote access

The UNIX user to be used in remote connection (i.e.: ansible) user and its SSH key file location need to be specified

  • Set in group_vars/all.yml file the UNIX user to be used by Ansible in the remote connection (default value ansible)

  • Modify ansible.cfg file to include the path to the SSH key of the ansible user used in remote connections (private-file-key variable)

    # SSH key
private_key_file = $HOME/ansible-ssh-key.pem

  • Modify all.yml file to include your ansible remote UNIX user (ansible_user variable) and

Configuring Ansible Playbooks

Encrypting secrets/key variables

All secrets/key/passwords variables are stored in a dedicated file, vars/vault.yml, so this file can be encrypted using Ansible Vault

vault.yml file is a Ansible vars file containing just a unique yaml variable, vault: a yaml dictionary containing all keys/passwords used by the different cluster components.

vault.yml sample file is like this:

---
# Encrypted variables - Ansible Vault
vault:
  # K3s secrets
  k3s:
    k3s_token: s1cret0
  # traefik secrets
  traefik:
    basic_auth_passwd: s1cret0
  # Minio S3 secrets
  minio:
    root_password: supers1cret0
    longhorn_key: supers1cret0
    velero_key: supers1cret0
    restic_key: supers1cret0
  # elastic search
....

All needed password-type variables used by the Playbooks are in the sample file var/picluster-vault.yml. This file is not encrypted and must be used to start the ansible setup.

The steps to configure passwords/keys used in all Playbooks is the following:

  1. Copy sample yaml var/picluster-vault.yml file and rename it as var/vault.yml

  2. Edit content of the file specifying your own values for each of the key/password/secret specified.

  3. Encrypt file using ansible-vault

    ansible-vault encrypt vault.yml

    The command ask for a ansible vault password to encrypt the file. After executing the command the file vault.yml is encrypted. Yaml content file is not readable.

Modify Ansible Playbook variables

Adjust ansible playbooks/roles variables defined within group_vars, host_vars and vars directories to meet your specific configuration.

The following table shows the variable files defined at ansible’s group and host levels

Group/Host Variable file Nodes affected
group_vars/all.yml all nodes of cluster + gateway node + pimaster
group_vars/control.yml control group: gateway node + pimaster
group_vars/k3s_cluster.yml all nodes of the k3s cluster
group_vars/k3s_master.yml K3s master nodes
host_vars/gateway.yml gateway node specific variables

The following table shows the variable files used for configuring the storage, backup server and K3S cluster and services.

Specific Variable File Configuration
vars/picluster.yml K3S cluster and services configuration variables
vars/dedicated_disks/local_storage.yml Configuration nodes local storage: Dedicated disks setup
vars/centralized_san/centralized_san_target.yml Configuration iSCSI target local storage and LUNs: Centralized SAN setup
vars/centralized_san/centralized_san_initiator.yml Configuration iSCSI Initiator: Centralized SAN setup
vars/backup/s3_minio.yml Configuration S3 Minio server

Installing the nodes

Update Raspberry Pi firmware

Update firmware in all Raspberry-PIs following the procedure described in “Raspberry PI firmware update”

Install gateway node

Install gateway Operating System on Rapberry PI.

The installation procedure followed is the described in “Ubuntu OS Installation” using cloud-init configuration files (user-data and network-config) for gateway, depending on the storage setup selected:

Storage Configuration User data Network configuration
Dedicated Disks user-data network-config
Centralized SAN user-data network-config

Configure gateway node

For automatically execute basic OS setup tasks and configuration of gateway’s services (DNS, DHCP, NTP, Firewall, etc.), executes the playbook:

ansible-playbook setup_picluster.yml --tags "gateway" [--ask-vault-pass]

Install cluster nodes.

Once gateway is up and running the rest of the nodes can be installed and connected to the LAN switch, so they can obtain automatic network configuration via DHCP.

Install node1-5 Operating System on Raspberry Pi

Follow the installation procedure indicated in “Ubuntu OS Installation” using the corresponding cloud-init configuration files (user-data and network-config) depending on the storage setup selected. Since DHCP is used there is no need to change default /boot/network-config file located in the ubuntu image.

Storage Architeture node1 node2 node3 node4 node5
Dedicated Disks user-data user-data user-data user-data user-data
Centralized SAN user-data user-data user-data user-data user-data

Configure cluster nodes

For automatically execute basic OS setup tasks (DNS, DHCP, NTP, etc.), executes the playbook:

ansible-playbook setup_picluster.yml --tags "node"

Configuring backup server (S3) and OS level backup

Configure backup server (Playbook assumes S3 server is installed in node1) and automated backup tasks at OS level with restic in all nodes (node1-node5 and gateway) running the playbook:

ansible-playbook backup_configuration.yml

K3S

K3S Installation

To install K3S cluster execute the playbook:

ansible-playbook k3s_install.yml

K3S basic services deployment

To deploy and configure basic services (metallb, traefik, certmanager, linkerd, longhorn, EFK, Prometheus, Velero) run the playbook:

ansible-playbook k3s_deploy.yml

Different ansible tags can be used to select the componentes to deploy:

ansible-playbook k3s_deploy.yml --tags <ansible_tag>

The following table shows the different components and their dependencies.

Ansible Tag Component to configure/deploy Dependencies
metallb Metal LB -
certmanager Cert-manager -
linkerd Linkerd Cert-manager
traefik Traefik Linkerd
longhorn Longhorn Linkerd
monitoring Prometheus Stack Longhorn, Linkerd
linkerd-viz Linkerd Viz Prometheus Stack, Linkerd
logging EFK Stack Longhorn, Linkerd
backup Velero Linkerd

K3s Cluster reset

If you mess anything up in your Kubernetes cluster, and want to start fresh, the K3s Ansible playbook includes a reset playbook, that you can use to remove the installation of K3S:

ansible-playbook k3s_reset.yml

Shutting down the Raspeberry Pi Cluster

To automatically shut down the Raspberry PI cluster, Ansible can be used.

For shutting down the cluster run this command:

ansible-playbook shutdown.yml

This playbook will connect to each Raspberry PI in the cluster (including gateway node) and execute the command sudo shutdown -h 1m, commanding the raspberry-pi to shutdown in 1 minute.

After a couple of minutes all raspberry pi will be shutdown. You can notice that when the Switch ethernet ports LEDs are off. Then it is safe to unplug the Raspberry PIs.

Updating Ubuntu packages

To automatically update Ubuntu OS packages run the following playbook:

ansible-playbook update.yml

This playbook automatically updates OS packages to the latest stable version and it performs a system reboot if needed.

Last Update: Oct 02, 2022

Comments: