Kubernetes Pi Cluster relase v1.6

Jan 29, 2023 • ricsanfre

Today I am pleased to announce the sixth release of Kubernetes Pi Cluster project (v1.6).

Main features/enhancements of this release are:

GitOps methodology adoption (ArgoCD)

In previous releases Ansible were used not only to configure cluster nodes OS and install K3S, but also to deploy Kubernetes applications in an imperative way, through the sequiential execution of installation/configuration commands (helm install and kubectl apply -f) contained in Ansible playbooks/roles and all helm chart values files and manifests needed to be applied in form of jinja2 templates.

In this new release, deployment of Kuberentes applications is completely managed by Argo CD, being Git repository the single source of truth for helm charts configuration files and kubernetes manifest. Kubernetes applications keep synch with manifest files stored in Git repository, enabling the implementation of a Continuous Delivery (CD) pipeline.


The automation source code has been completely refactored:

  • New packaged Kubernetes applications, in form of helm charts or sets of manifest files, have been developed, so they can be deployed with ArgoCD.
  • Changes in Ansible automation code to automatically boot the cluster using ArgoCD
  • Remove old ansible code to deploy Kuberentes applications

Check further details about ArgoCD installation and configuration in the documentation and the new Quick Start Instructions.

New Secrets Management solution (Hashicorp Vault)

Related to the previous feature, a Secret Management tool has been integrated in the cluster to maanage the creation of secrets needed by the Cluster Applications.

HashiCorp Vault is used now as Secret Management solution for Raspberry PI cluster. All cluster secrets (users, passwords, api tokens, etc) are securely encrypted and stored in Vault.

Vault is deployed as a external service, not runing as Kuberentes application. External Secrets Operator is used to automatically generate the Kubernetes Secrets from Vault data that Kubernetes applications could need.


Cluster bootstrap process, using Ansible, takes care of the installation and configuration of Hashicorp Vault and the initial load of secrets needed by the cluster applications.

Check further details in Vautl Installation doc.

From Monitoring to Observability platform

Move from a Monitoring platform, based on Prometheus (metrics) and EFK (logs) to a Observability Platform adding traces monitoring and a single plane of glass.

New observability solution based on Loki (logs), Tempo (traces), Prometheus (metrics) and Grafana as single plane of glass for monitoring.


Main features:

  • Grafana Loki as complement of the available EFK platform, not a replacement. ES is used mainly for Log Analytics (log content is completely indexed) while Loki can be used for Observability (only log labels are indexed) having together logs, metrics and traces in the same Grafana Dashboards.

  • Common logs collection/distrution layer based on fluentbit/fluentd is used to feed logs to ES and to Loki, instead of deploying a separate collector (Loki promtail)

  • Grafana Tempo as distributed tacing solution integrating Linkerd distributed tracing capability.

Upgrade software components to latest stable version

Type Software Latest Version tested Notes
OS Ubuntu 20.04.3 OS need to be tweaked for Raspberry PI when booting from external USB
Control Ansible 2.12.1  
Control cloud-init 21.4 version pre-integrated into Ubuntu 20.04
Kubernetes K3S v1.24.7 K3S version
Kubernetes Helm v3.6.3  
Metrics Kubernetes Metrics Server v0.6.1 version pre-integrated into K3S
Computing containerd v1.6.8-k3s1 version pre-integrated into K3S
Networking Flannel v0.19.2 version pre-integrated into K3S
Networking CoreDNS v1.9.1 version pre-integrated into K3S
Networking Metal LB v0.13.7 Helm chart version: 0.13.7
Service Mesh Linkerd v2.12.2 Helm chart version: linkerd-control-plane-1.9.4
Service Proxy Traefik v2.9.1 Helm chart version: 18.1.0
Storage Longhorn v1.3.2 Helm chart version: 1.3.2
TLS Certificates Certmanager v1.10.0 Helm chart version: v1.10.0
Logging ECK Operator 2.4.0 Helm chart version: 2.4.0
Logging Elastic Search 8.1.2 Deployed with ECK Operator
Logging Kibana 8.1.2 Deployed with ECK Operator
Logging Fluentbit 2.0.4 Helm chart version: 0.21.0
Logging Fluentd 1.15.2 Helm chart version: 0.3.9. Custom docker image from official v1.15.2
Logging Loki 2.6.1 Helm chart grafana/loki version: 3.3.0
Monitoring Kube Prometheus Stack 0.61.1 Helm chart version: 43.3.1
Monitoring Prometheus Operator 0.61.1 Installed by Kube Prometheus Stack. Helm chart version: 43.3.1
Monitoring Prometheus 2.40.5 Installed by Kube Prometheus Stack. Helm chart version: 43.3.1
Monitoring AlertManager 0.25.0 Installed by Kube Prometheus Stack. Helm chart version: 43.3.1
Monitoring Grafana 9.3.1 Helm chart version grafana-6.48.2. Installed as dependency of Kube Prometheus Stack chart. Helm chart version: 43.3.1
Monitoring Prometheus Node Exporter 1.5.0 Helm chart version: prometheus-node-exporter-4.8.2. Installed as dependency of Kube Prometheus Stack chart. Helm chart version: 43.3.1
Monitoring Prometheus Elasticsearch Exporter 1.5.0 Helm chart version: prometheus-elasticsearch-exporter-4.15.1
Backup Minio RELEASE.2022-09-22T18-57-27Z  
Backup Restic 0.12.1  
Backup Velero 1.9.3 Helm chart version: 2.32.1
Secrets Hashicorp Vault 1.12.2  
Secrets External Secret Operator 0.7.1 Helm chart version: 0.7.1
GitOps Argo CD v2.5.6 Helm chart version: 5.17.1

Release v1.6.0 Notes

Apply GitOps methodology using ArgoCD to deploy and manage Kubernetes Applications, integrate Hashicorp Vault secret management solution and transform monitoring platform into observability platform (logs, traces and metrics monitoring).

Release Scope:

  • GitOps methodology
    • Argo CD deployment
    • New packaged Kubernetes applications (helm charts and manifest files) to be deployed using ArgoCD
    • Automate cluster bootstraping with ArgoCD using Ansible
    • Ansible playbooks/roles/vars refactoring
  • Integrate Secrets Management solution
    • Hashicorp Vault deployment
    • Kuberentes authorization mechanism integration
    • External Secrets Operator deployment
  • Observability platform
    • Grafana Loki and Grafana Tempo deployment
    • Grafana as cluster operations single pane of glass
    • Fluentbit/Fluentd configuration to distribute logs to ES and Loki
    • Linkerd distributed tracing integration
    • Traefik tracing integration and automatic correlation with access logs
  • Automation enhancements
    • Integration of Ansible vault and GPG to automate the encrypt/decrypt process
    • Automatic generation of credentials and load in Vault
    • Add Makefile