Kubernetes Pi Cluster relase v1.4
Aug 4, 2022 • ricsanfre
Today I am pleased to announce the fourth release of Kubernetes Pi Cluster project (v1.4).
Main features/enhancements of this release are:
Logging solution (EFK) enhancements
-
Logging collection and distribution architecture has been completely redesigned.
Previous logging collection and distribution architecture was based on
fluent-bit
’s logs forwarders installed in each cluster node, collecting containers logs and ingesting them into Elastisearch (Forwarders-only architecture). This architecture has been replaced by a Forwarders/Aggregator architecture. This new architecture, introduces a Log Aggregation layer, based onfluentd
, between the collectors (forwarders) and the backend (Elasticsearch).With this architecture:
- In the forwarder layer, lightweight logging agent instances (fluentbit) do minimal processing (collects and parsing) and then use the
forward protocol
(propietary Fluentbit/Fluentd protocol). - In the aggregation layer, a much heavier logging instance (fluentd) may perform more filtering and processing before routing to the appropriate backend. In this case, logs can be routed to more than one backed. The destination of all logs can be Elastisearch database (default route) and some logs can be routed to a different backend for further processing. For example Kafka can be deployed as backend to build a Data Streaming Analytics architecture (Kafka, Apache Spark, Flink, etc) and route only the logs from specfic applications.
- In the forwarder layer, lightweight logging agent instances (fluentbit) do minimal processing (collects and parsing) and then use the
-
Logging solution integrated in Prometheus
Fluentbit’s log forwarders and Fluentd’s log aggregator have been configured to export its metrics, including fluentbit buffer storage metrics. Prometheus Elasticsearch Exporter is deployed in order to obtain ES metrics.
-
Kibana and Elasticsearch have been upgraded to release 8.1.2
-
Logging parsing enhancements and log fields standardization
Standardize logs fields names to avoid ES ingestion errors and activate fluentbit’s built-in capability to process JSON messages payloads and Multiline logs.
Monitor external nodes in Prometheues
Leveraging fluent-bit built-in node-exporter-metric and prometheus-exporter plugins, external nodes (i.e: gateway
) can expose basic metrics that can be ingested into Prometheus.
gateway
’s fluentbit agent, already deployed to collect its logs, has been configured to expose Prometheus metrics. Since the metrics are similar to the ones generated by Prometheus node exporter, these metrics can be integrated in the same Grafana Dashboards.
Configuration of new Kubernetes’ graceful node shutdown feature
Since Kubernetes 1.21 there is available Graceful node shutdown feature. This feature, still in beta status, is documented here. and it ensures that pods follow the normal pod termination process during the node shutdown.
My home cluster is not always running, I usually shutdown my cluster once I finish my testing/learning. Before applying this feature, when shutting down my cluster, pods did not shutdown gracefully and the cluster sometimes remained in an inconsistent status. It forced me to reset the whole installation. Also shutting-down process took long time to be completed (15-30 min).
Since the feature has been applied, pods are gracefully shutdown and cluster remains in consistent status, and shutdown process is faster (<5 min).
Upgrade software components to latest stable version
Type | Software | Latest Version tested | Notes |
---|---|---|---|
OS | Ubuntu | 20.04.3 | OS need to be tweaked for Raspberry PI when booting from external USB |
Control | Ansible | 2.12.1 | |
Control | cloud-init | 21.4 | version pre-integrated into Ubuntu 20.04 |
Kubernetes | K3S | v1.24.3 | K3S version |
Kubernetes | Helm | v3.6.3 | |
Metrics | Kubernetes Metrics Server | v0.5.2 | version pre-integrated into K3S |
Computing | containerd | v1.5.13-k3s1 | version pre-integrated into K3S |
Networking | Flannel | v0.18.1 | version pre-integrated into K3S |
Networking | CoreDNS | v1.9.1 | version pre-integrated into K3S |
Networking | Metal LB | v0.13.4 | Helm chart version: metallb-0.13.4 |
Service Mesh | Linkerd | v2.11.4 | Helm chart version: linkerd2-2.11.4 |
Service Proxy | Traefik | v2.6.2 | Helm chart: traefik-10.14.100 version pre-integrated into K3S |
Storage | Longhorn | v1.3.0 | Helm chart version: longhorn-1.3.0 |
SSL Certificates | Certmanager | v1.9.1 | Helm chart version: cert-manager-v1.9.1 |
Logging | ECK Operator | 2.3.0 | Helm chart version: eck-operator-2.3.0 |
Logging | Elastic Search | 8.1.2 | Deployed with ECK Operator |
Logging | Kibana | 8.1.2 | Deployed with ECK Operator |
Logging | Fluentbit | 1.9.6 | Helm chart version: fluent-bit-0.20.4 |
Logging | Fluentd | 1.14 | Helm chart version: 0.3.9. Custom docker image from official v1.14 |
Monitoring | Kube Prometheus Stack | 0.58.0 | Helm chart version: kube-prometheus-stack-39.1.0 |
Monitoring | Prometheus Operator | 0.58.0 | Installed by Kube Prometheus Stack. Helm chart version: kube-prometheus-stack-39.1.0 |
Monitoring | Prometheus | 2.37 | Installed by Kube Prometheus Stack. Helm chart version: kube-prometheus-stack-39.1.0 |
Monitoring | AlertManager | 0.24 | Installed by Kube Prometheus Stack. Helm chart version: kube-prometheus-stack-39.1.0 |
Monitoring | Grafana | 9.0.4 | Helm chart version grafana-6.32.9. Installed as dependency of Kube Prometheus Stack chart. Helm chart version: kube-prometheus-stack-39.1.0 |
Monitoring | Prometheus Node Exporter | 1.3.1 | Helm chart version: prometheus-node-exporter-3.3.1. Installed as dependency of Kube Prometheus Stack chart. Helm chart version: kube-prometheus-stack-39.1.0 |
Monitoring | Prometheus Elasticsearch Exporter | 1.3.0 | Helm chart version: prometheus-elasticsearch-exporter-4.13.0 |
Backup | Minio | RELEASE.2022-05-04T07-45-27Z | |
Backup | Restic | 0.12.1 | |
Backup | Velero | 1.9.0 | Helm chart version: velero-2.30.1 |
Release v1.4.0 Notes
Re-architecting logging solution and configure Kubernetes’ graceful node shutdown.
Release Scope:
- Logging solution (EFK) enhancements
- Upgrade Elasticsearch and Kibana to v8.1
- Logging collection and distribution architecture based on fluentd/fluentbit re-architected
- Prometheus integration: Adding Elasticsearch and Fluentbit/Fluentd metrics
- Logging parsing enhancements and log fields standardization
- Configure Kubernetes’ graceful node shutdown feature
- Monitor external nodes (
gateway
) in Prometheues using fluent-bit agent. - Upgrade software components to latest stable version
Comments:
- Older
- Newer