Pi Cluster Documentation

K3S Installation

K3S is a lightweight kubernetes built for IoT and edge computing, provided by the company Rancher. The following picture shows the K3S architecture (source K3S).

K3S Architecture

In K3S all kubernetes processes are consolidated within one single binary. The binary is deployed on servers with two different k3s roles (k3s-server or k3s-agent).

  • k3s-server: starts all kubernetes control plane processes (API, Scheduler and Controller) and worker proceses (Kubelet and kube-proxy), so master node can be used also as worker node.
  • k3s-agent: consolidating all kuberentes worker processes (Kubelet and kube-proxy).

Control-plane nodes will be configured so no load is deployed in it.

Nodes preconfiguration

  • Step 1: Enable iptables to see bridged traffic

    Load br_netfilter kernel module an modify settings to let iptables see bridged traffic

      cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
  br_netfilter
  EOF
    
  cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
  net.bridge.bridge-nf-call-ip6tables = 1
  net.bridge.bridge-nf-call-iptables = 1
  EOF

  sudo sysctl --system

  • Step 2: Disable swap memory (only x86 nodes)

    sudo swapoff -a

    Modify /etc/fstab to make this change permanent, commenting line corresponding to swap.

  • Step 3: Enable cgroup on Raspberry PI nodes.

    Modify file /boot/firmware/cmdline.txt to include the line:

    cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

  • Step 4: Reboot the server

Single-server Setup with an Embedded DB

In this case a single node will be configured as master node. K3s embedded sqlite database is used in this case.

In this configuration, each agent node is registered to the same server node. A K3s user can manipulate Kubernetes resources by calling the K3s API on the server node.

K3S Architecture

Master node installation

  • Step 1: Prepare K3S kubelet configuration file

    Create file /etc/rancher/k3s/kubelet.config

    apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
shutdownGracePeriod: 30s
shutdownGracePeriodCriticalPods: 10s

    This kubelet configuration enables new kubernetes feature Graceful node shutdown. This feature is available since Kuberentes 1.21, it is still in beta status, and it ensures that pods follow the normal pod termination process during the node shutdown.

    See further details in “Kuberentes documentation: Graceful-shutdown”.

  • Step 2: Installing K3S control plane node

    For installing the master node execute the following command:

      curl -sfL https://get.k3s.io | K3S_TOKEN=<server_token> sh -s - server --write-kubeconfig-mode '0644' --node-taint 'node-role.kubernetes.io/control-plane:NoSchedule' --disable 'servicelb' --disable 'traefik' --disable 'local-storage' --kube-controller-manager-arg 'bind-address=0.0.0.0' --kube-proxy-arg 'metrics-bind-address=0.0.0.0' --kube-scheduler-arg 'bind-address=0.0.0.0' --kubelet-arg 'config=/etc/rancher/k3s/kubelet.config' --kube-controller-manager-arg 'terminated-pod-gc-threshold=10'

    Where:

    • server_token is shared secret within the cluster for allowing connection of worker nodes
    • --write-kubeconfig-mode '0644' gives read permissions to kubeconfig file located in /etc/rancher/k3s/k3s.yaml
    • --node-taint 'node-role.kubernetes.io/control-plane:NoSchedule' makes master node not schedulable to run any pod. Only pods marked with specific tolerance will be scheduled on master node.
    • --disable servicelb to disable default service load balancer installed by K3S (Klipper Load Balancer). Metallb will be used instead.
    • --disable local-storage to disable local storage persistent volumes provider installed by K3S (local-path-provisioner). Longhorn will be used instead
    • --disable traefik to disable default ingress controller installed by K3S (Traefik). Traefik will be installed from helm chart.
    • --kube-controller-manager.arg, --kube-scheduler-arg and --kube-proxy-arg to bind those components not only to 127.0.0.1 and enable metrics scraping from a external node.
    • --kubelet-arg 'config=/etc/rancher/k3s/kubelet.config' provides kubelet configuraion parameters. See Kubernetes Doc: Kubelet Config File
    • --kube-controller-manager-arg 'terminated-pod-gc-threshold=10'. Setting limit to 10 terminated pods that can exist before the terminated pod garbage collector starts deleting terminated pods. See Kubernetes Doc: Pod Garbage collection


  • Step 3: Install Helm utility

    Kubectl is installed as part of the k3s server installation (/usr/local/bin/kubectl), but helm need to be installed following this instructions.

  • Step 4: Copy k3s configuration file to Kubernets default directory ($HOME/.kube/config), so kubectl and helm utilities can find the way to connect to the cluster.

     mkdir $HOME/.kube
 cp /etc/rancher/k3s/k3s.yaml $HOME/.kube/.

Workers installation

  • Step 1: Prepare K3S kubelet configuration file

    Create file /etc/rancher/k3s/kubelet.config

    apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
shutdownGracePeriod: 30s
shutdownGracePeriodCriticalPods: 10s

  • Step 2: Installing K3S worker node

    For installing the master node execute the following command:

    curl -sfL https://get.k3s.io | K3S_URL='https://<k3s_master_ip>:6443' K3S_TOKEN=<server_token> sh -s - --node-label 'node_type=worker' --kubelet-arg 'config=/etc/rancher/k3s/kubelet.config' --kube-proxy-arg 'metrics-bind-address=0.0.0.0'

    Where:

    • server_token is shared secret within the cluster for allowing connection of worker nodes
    • k3s_master_ip is the k3s master node ip
    • --node-label 'node_type=worker' add a custom label node_type to the worker node.
    • --kubelet-arg 'config=/etc/rancher/k3s/kubelet.config' provides kubelet configuraion parameters. See Kubernetes Doc: Kubelet Config File
    • --kube-proxy-arg 'metrics-bind-address=0.0.0.0' to enable kube-proxy metrics scraping from a external node


  • Step 3: Specify role label for worker nodes

    From master node, assign a role label to worker nodes, so when executing kubectl get nodes command ROLE column show worker role for workers nodes.

    kubectl label nodes <worker_node_name> kubernetes.io/role=worker

High-Availability K3s

Three or more server nodes that will serve the Kubernetes API and run other control plane services An embedded etcd datastore (as opposed to the embedded SQLite datastore used in single-server setups).

A load balancer is needed for providing High availability to Kubernetes API. In this case, a network load balancer, HAProxy , will be used.

K3S Architecture

Load Balancer (HAProxy)

HAProxy need to be installed in one node. If it is possible select a node which is not part of the K3s cluster. In my case I will install it on node1.

To install and configure HAProxy:

  • Step 1. Install haproxy

    sudo apt install haproxy

  • Step 2. Configure haproxy

    Edit file /etc/haproxy/haproxy.cfg

    global
  log /dev/log  local0
  log /dev/log  local1 notice
  chroot /var/lib/haproxy
  stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
  stats timeout 30s
  user haproxy
  group haproxy
  daemon

defaults
  log global
  mode http
  option httplog
  option dontlognull
  retries 3
  timeout http-request 10s
  timeout queue 20s
  timeout connect 10s
  timeout client 1h
  timeout server 1h
  timeout http-keep-alive 10s
  timeout check 10s
  errorfile 400 /etc/haproxy/errors/400.http
  errorfile 403 /etc/haproxy/errors/403.http
  errorfile 408 /etc/haproxy/errors/408.http
  errorfile 500 /etc/haproxy/errors/500.http
  errorfile 502 /etc/haproxy/errors/502.http
  errorfile 503 /etc/haproxy/errors/503.http
  errorfile 504 /etc/haproxy/errors/504.http


#---------------------------------------------------------------------
# apiserver frontend which proxys to the control plane nodes
#---------------------------------------------------------------------
frontend k8s_apiserver
    bind *:6443
    mode tcp
    option tcplog
    default_backend k8s_controlplane

#---------------------------------------------------------------------
# round robin balancing for apiserver
#---------------------------------------------------------------------
backend k8s_controlplane
    option httpchk GET /healthz
    http-check expect status 200
    mode tcp
    option ssl-hello-chk
    balance     roundrobin
      server node2 10.0.0.12:6443 check
      server node3 10.0.0.13:6443 check
      server node4 10.0.0.14:6443 check

    With this configuration haproxy will balance requests to API server (TCP port 6443), following a round-robin balancing method, between the 3 master nodes configured.

    IP address to be used for kubernetes API, will be gateway’s IP address.

  • Step 3: Restart HAProxy

    sudo systemctl restart haproxy

  • Step 4: Enable haproxy to boot

    systemctl enable haproxy

Master nodes installation

Embedded etcd data store will be used. Installation procedure is described in K3S documentation: High Availability Embedded etcd.

  • Step 1: Create config directory

    sudo mkdir -p /etc/rancher/k3s

  • Step 2: Create token file in all nodes

    Create file /etc/rancher/k3s/cluster-token containing token value. K3s token is a shared secret among all nodes of the cluster (master and worker nodes)

    Instead of using K3S_TOKEN environment variable during installation, --token-file argument will be used.

    echo "supersecrettoken" > /etc/rancher/k3s/cluster-token

  • Step 3: Prepare K3S kubelet configuration file.

    Create file /etc/rancher/k3s/kubelet.config

    apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
shutdownGracePeriod: 30s
shutdownGracePeriodCriticalPods: 10s

    This kubelet configuration enables new kubernetes feature Graceful node shutdown. This has been explained in the previous section: single master node installation.

  • Step 4: Prepare K3s config file

    Create file /etc/rancher/k3s/config.yaml containing all configuration options needed. They are equivalent to the K3s arguments

    token-file: /etc/rancher/k3s/cluster-token
disable:
- local-storage
- servicelb
- traefik
etcd-expose-metrics: true
kube-controller-manager-arg:
- bind-address=0.0.0.0
- terminated-pod-gc-threshold=10
kube-proxy-arg:
- metrics-bind-address=0.0.0.0
kube-scheduler-arg:
- bind-address=0.0.0.0
kubelet-arg:
- config=/etc/rancher/k3s/kubelet.config
node-taint:
- node-role.kubernetes.io/master=true:NoSchedule
tls-san:
- 10.0.0.11
write-kubeconfig-mode: 644

    This configuration is equivalent to the following k3s arguments:

    --toke-file /etc/rancher/k3s/cluster-token
--write-kubeconfig-mode '0644'
--disable 'servicelb'
--disable 'traefik'
--disable 'local-storage'
--node-taint 'node-role.kubernetes.io/master=true:NoSchedule'
--etcd-expose-metrics
--tls-san 10.0.0.11
--kube-controller-manager-arg 'bind-address=0.0.0.0'
--kube-proxy-arg 'metrics-bind-address=0.0.0.0'
--kube-scheduler-arg 'bind-address=0.0.0.0'
--kubelet-arg 'config=/etc/rancher/k3s/kubelet.config'
--kube-controller-manager-arg 'terminated-pod-gc-threshold=10'

    Parameters are the same which have been configured during installation in single master node deployment, adding the following:

    • token-file parameter instead K3S_TOKEN environment variable
    • tls-san parameter to add k3s api load balancer ip as Subject Alternative Names on TLS cert created by K3S.
    • etcd-expose-metrics to expose etcd metrics

  • Step 5. Install primary master node

    curl -sfL https://get.k3s.io | sh -s - server --cluster-init

  • Step 6. Install secondary master nodes

    curl -sfL https://get.k3s.io | sh -s - server --server https://<ip or hostname of first master node>:6443

Worker nodes installation

  • Step 1: Create config directory

    sudo mkdir -p /etc/rancher/k3s

  • Step 2: Create token file in all nodes

    Create file /etc/rancher/k3s/cluster-token containing token value. K3s token is a shared secret among all nodes of the cluster (master and worker nodes)

    Instead of using K3S_TOKEN environment variable during installation, --token-file argument will be used.

    echo "supersecrettoken" > /etc/rancher/k3s/cluster-token

  • Step 3: Prepare K3S kubelet configuration file.

    Create file /etc/rancher/k3s/kubelet.config

    apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
shutdownGracePeriod: 30s
shutdownGracePeriodCriticalPods: 10s

  • Step 4: Prepare K3s config file

    Create file /etc/rancher/k3s/config.yaml containing all configuration options needed.

    token-file: /etc/rancher/k3s/cluster-token
node-label:
  - 'node_type=worker'
kubelet-arg:
  - 'config=/etc/rancher/k3s/kubelet.config'
kube-proxy-arg:
  - 'metrics-bind-address=0.0.0.0'

    This configuration is equivalent to the following k3s arguments:

    --toke-file /etc/rancher/k3s/cluster-token
--node-label 'node_type=worker'
--kubelet-arg 'config=/etc/rancher/k3s/kubelet.config'
--kube-proxy-arg 'metrics-bind-address=0.0.0.0'

  • Step 5. Install agent node

    curl -sfL https://get.k3s.io | sh -s - agent --server https://<k3s_api_loadbalancer_ip>:6443

Installing custom CNI

By default K3S install Flannel as CNI. If other CNI is going to be used default Flannel CNI need to be disabled during installation.

See details in K3S Networking - Use custom CNI

K3S master nodes need to be installed with the following additional options:

  • --flannel-backend=none: to disable Fannel instalation
  • --disable-network-policy: Most CNI plugins come with their own network policy engine, so it is recommended to set –disable-network-policy as well to avoid conflicts.

K3S Auto-deployed Add-ons

K3s provides the capability to automatically deploy manifest files (AddOns). On server nodes, any file found in /var/lib/rancher/k3s/server/manifests is automatically deployed to Kubernetes in a manner similar to kubectl apply command, both on startup and when the file is changed on disk. Deleting files out of this directory will not delete the corresponding resources from the cluster.

K3s comes with a number of packaged components that are deployed as AddOns via that manifests directory: coredns, traefik, local-storage, and metrics-server.

Manifests are tracked as AddOn custom resources (CRD) in the kube-system namespace.

Installation of this addOns can be disabled during k3s installation adding:

  • --disable '<addon>': Where <addon> can be coredns, traefik, local-storage or metric-server

K3s includes also a built-in Helm Controller that manages installing, upgrading/reconfiguring, and uninstalling Helm charts using a HelmChart Custom Resource Definition (CRD). Paired with auto-deploying AddOn manifests, installing a Helm chart can be automated by creating a single manifiest file on /var/lib/rancher/k3s/server/manifests.

K3s uses this built-in helm chart controller only to deploy traefik. Rest of add-ons are instralled using Kubernetes manifest files.

HelmChart controller can be disabled to avoid conflicts with other controllers (i.e.: Helm Controller deployed by GitOps solution FluxCD) and all the add-ons can be installed manually, following same installation process of any other K8S distribution.

To disable K3s HelmChart Controller the following additional installation option need to be added:

  • --disable-helm-controller: to disable K3s helm controller

If HelmChart controller is disabled Traefik add-ons need to be disabled as well

  • --disable 'traefik': to disable Traefik installation

See further details in K3s documentationt - Managing k3s packaged components

Remote Access

To enable remote access to the cluster using kubectl and helm applications follow the following procedure

  • Step 1: Install helm and kubectl utilities

  • Step 2: Copy k3s configuration file, located in /etc/rancher/k3s/k3s.yaml, to $HOME/.kube/config.

  • Step 3: Modify k3s.yaml configuration file for using the IP address instead of localhost

  • Step 4: Enable HTTPS connectivity on port 6443 between the server and the k3s control node

In case of HA deployment, k3s api load balancer ip can be used instead of the IP of any of the single nodes.

K3S Automatic Upgrade

K3s cluster upgrade can be automated using Rancher’s system-upgrade-controller. This controller uses a custom resource definition (CRD), Plan, to schedule upgrades based on the configured plans

See more details in K3S Automated Upgrades documentation

  • Step 1. Install Rancher’s system-upgrade-controller

    kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml

  • Step 2. Configure upgrade plans

    At least two upgrade plans need to be configured: a plan for upgrading server (master) nodes and a plan for upgrading agent (worker) nodes.

    Plan for master: k3s-master-upgrade.yml

    apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: k3s-server
  namespace: system-upgrade
  labels:
    k3s-upgrade: server
spec:
  nodeSelector:
    matchExpressions:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
  serviceAccountName: system-upgrade
  concurrency: 1
  # Cordon node before upgrade it
  cordon: true
  upgrade:
    image: rancher/k3s-upgrade
  version: <new_version>

    Plan for worker: k3s-agent-upgrade.yml

    apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: k3s-agent
  namespace: system-upgrade
  labels:
    k3s-upgrade: agent
spec:
  nodeSelector:
    matchExpressions:
      - key: node-role.kubernetes.io/control-plane
        operator: DoesNotExist
  serviceAccountName: system-upgrade
  # Wait for k3s-server upgrade plan to complete before executing k3s-agent plan
  prepare:
    image: rancher/k3s-upgrade
    args:
      - prepare
      - k3s-server
  concurrency: 1
  # Cordon node before upgrade it
  cordon: true
  upgrade:
    image: rancher/k3s-upgrade
  version: <new_version>

  • Step 3. Execute upgrade plans

    kubectl apply -f k3s-server-upgrade.yml k3s-agent-upgrade.yml

Reset the cluster

To reset the cluster execute k3s uninstall script in master and worker nodes

On each worker node, execute:

/usr/local/bin/k3s-agent-uninstall.sh

On each master node, execute

/usr/local/bin/k3s-uninstall.sh

Ansible Automation

K3s cluster installation and reset procedures have been automated with Asible playbooks

For installing the cluster execute:

ansible-playbook k3s_install.yml

For resetting the cluster execute:

ansible-playbook k3s_reset.yml

Configure master node to enable remote deployment of pods with Ansible

Ansible collection for managing kubernetes cluster is available: kubernetes.core ansible collection.

For using this ansible collection from the pimaster node, python package kubernetes need to be installed on k3s master node

pip3 install kubernetes
Last Update: Sep 28, 2024

Comments: