Kubernetes Pi Cluster relase v1.4

Aug 4, 2022 • ricsanfre

Today I am pleased to announce the fourth release of Kubernetes Pi Cluster project (v1.4).

Main features/enhancements of this release are:

Logging solution (EFK) enhancements

  • Logging collection and distribution architecture has been completely redesigned.

    Previous logging collection and distribution architecture was based on fluent-bit’s logs forwarders installed in each cluster node, collecting containers logs and ingesting them into Elastisearch (Forwarders-only architecture). This architecture has been replaced by a Forwarders/Aggregator architecture. This new architecture, introduces a Log Aggregation layer, based on fluentd, between the collectors (forwarders) and the backend (Elasticsearch).

    K3S-EFK-Architecture

    With this architecture:

    • In the forwarder layer, lightweight logging agent instances (fluentbit) do minimal processing (collects and parsing) and then use the forward protocol (propietary Fluentbit/Fluentd protocol).
    • In the aggregation layer, a much heavier logging instance (fluentd) may perform more filtering and processing before routing to the appropriate backend. In this case, logs can be routed to more than one backed. The destination of all logs can be Elastisearch database (default route) and some logs can be routed to a different backend for further processing. For example Kafka can be deployed as backend to build a Data Streaming Analytics architecture (Kafka, Apache Spark, Flink, etc) and route only the logs from specfic applications.

  • Logging solution integrated in Prometheus

    Fluentbit’s log forwarders and Fluentd’s log aggregator have been configured to export its metrics, including fluentbit buffer storage metrics. Prometheus Elasticsearch Exporter is deployed in order to obtain ES metrics.

  • Kibana and Elasticsearch have been upgraded to release 8.1.2

  • Logging parsing enhancements and log fields standardization

    Standardize logs fields names to avoid ES ingestion errors and activate fluentbit’s built-in capability to process JSON messages payloads and Multiline logs.

Monitor external nodes in Prometheues

Leveraging fluent-bit built-in node-exporter-metric and prometheus-exporter plugins, external nodes (i.e: gateway) can expose basic metrics that can be ingested into Prometheus.

gateway’s fluentbit agent, already deployed to collect its logs, has been configured to expose Prometheus metrics. Since the metrics are similar to the ones generated by Prometheus node exporter, these metrics can be integrated in the same Grafana Dashboards.

Configuration of new Kubernetes’ graceful node shutdown feature

Since Kubernetes 1.21 there is available Graceful node shutdown feature. This feature, still in beta status, is documented here. and it ensures that pods follow the normal pod termination process during the node shutdown.

My home cluster is not always running, I usually shutdown my cluster once I finish my testing/learning. Before applying this feature, when shutting down my cluster, pods did not shutdown gracefully and the cluster sometimes remained in an inconsistent status. It forced me to reset the whole installation. Also shutting-down process took long time to be completed (15-30 min).

Since the feature has been applied, pods are gracefully shutdown and cluster remains in consistent status, and shutdown process is faster (<5 min).

Upgrade software components to latest stable version

Type Software Latest Version tested Notes
OS Ubuntu 20.04.3 OS need to be tweaked for Raspberry PI when booting from external USB
Control Ansible 2.12.1  
Control cloud-init 21.4 version pre-integrated into Ubuntu 20.04
Kubernetes K3S v1.24.3 K3S version
Kubernetes Helm v3.6.3  
Metrics Kubernetes Metrics Server v0.5.2 version pre-integrated into K3S
Computing containerd v1.5.13-k3s1 version pre-integrated into K3S
Networking Flannel v0.18.1 version pre-integrated into K3S
Networking CoreDNS v1.9.1 version pre-integrated into K3S
Networking Metal LB v0.13.4 Helm chart version: metallb-0.13.4
Service Mesh Linkerd v2.11.4 Helm chart version: linkerd2-2.11.4
Service Proxy Traefik v2.6.2 Helm chart: traefik-10.14.100 version pre-integrated into K3S
Storage Longhorn v1.3.0 Helm chart version: longhorn-1.3.0
SSL Certificates Certmanager v1.9.1 Helm chart version: cert-manager-v1.9.1
Logging ECK Operator 2.3.0 Helm chart version: eck-operator-2.3.0
Logging Elastic Search 8.1.2 Deployed with ECK Operator
Logging Kibana 8.1.2 Deployed with ECK Operator
Logging Fluentbit 1.9.6 Helm chart version: fluent-bit-0.20.4
Logging Fluentd 1.14 Helm chart version: 0.3.9. Custom docker image from official v1.14
Monitoring Kube Prometheus Stack 0.58.0 Helm chart version: kube-prometheus-stack-39.1.0
Monitoring Prometheus Operator 0.58.0 Installed by Kube Prometheus Stack. Helm chart version: kube-prometheus-stack-39.1.0
Monitoring Prometheus 2.37 Installed by Kube Prometheus Stack. Helm chart version: kube-prometheus-stack-39.1.0
Monitoring AlertManager 0.24 Installed by Kube Prometheus Stack. Helm chart version: kube-prometheus-stack-39.1.0
Monitoring Grafana 9.0.4 Helm chart version grafana-6.32.9. Installed as dependency of Kube Prometheus Stack chart. Helm chart version: kube-prometheus-stack-39.1.0
Monitoring Prometheus Node Exporter 1.3.1 Helm chart version: prometheus-node-exporter-3.3.1. Installed as dependency of Kube Prometheus Stack chart. Helm chart version: kube-prometheus-stack-39.1.0
Monitoring Prometheus Elasticsearch Exporter 1.3.0 Helm chart version: prometheus-elasticsearch-exporter-4.13.0
Backup Minio RELEASE.2022-05-04T07-45-27Z  
Backup Restic 0.12.1  
Backup Velero 1.9.0 Helm chart version: velero-2.30.1

Release v1.4.0 Notes

Re-architecting logging solution and configure Kubernetes’ graceful node shutdown.

Release Scope:

  • Logging solution (EFK) enhancements
    • Upgrade Elasticsearch and Kibana to v8.1
    • Logging collection and distribution architecture based on fluentd/fluentbit re-architected
    • Prometheus integration: Adding Elasticsearch and Fluentbit/Fluentd metrics
    • Logging parsing enhancements and log fields standardization
  • Configure Kubernetes’ graceful node shutdown feature
  • Monitor external nodes (gateway) in Prometheues using fluent-bit agent.
  • Upgrade software components to latest stable version

Comments: