Kubernetes development environment

The development setup is the following:

• A dedicated docker network is configured for the cluster
• k3d cluster installed disabling flannel CNI, kube-proxy and load balancer.
• Cilium is installed as CNI which also takes care of the routing which was handled by kube-proxy.
• Cilium L2-LB awareness is enabled, and a set of IP’s are configured for Loadbalancers services and advertised via L2 announcements.
• Docker cluster nodes and Load balancer Kubernetes services are reachable from local host through docker network

Automated workflow (recommended)

The automated workflow is implemented in dev/ and kubernetes/clusters/dev/. It creates the Docker network, the k3d cluster, the external RustFS/Vault services, and bootstraps Cilium, CoreDNS and Flux.

Quick start

cd dev
make dev-preflight   # tools + MemAvailable check (>= 8 GiB)
make dev-up          # network + cluster + external services + Flux + smoke
make dev-status      # context guard, Flux, HelmReleases, pods, warnings
make dev-smoke       # nodes, DNS, metrics, local-path PVC, base pods
make dev-usage       # host memory, docker stats, kubectl top
make dev-down        # stop external services and delete the cluster (data kept)
make dev-recreate    # down + up + smoke

dev-preflight refuses to continue if the host has less than 8 GiB available. Every Make target asserts the k3d-picluster kubectl context.

Base components

The k3s image defaults to rancher/k3s:v1.37.1-k3s1 to match prod. Override it with K3S_IMAGE=rancher/k3s:<tag> or add agents with K3D_AGENTS=1 for multi-node scheduling tests.

Profiles

Profiles are opt-in. Add one file per profile to kubernetes/clusters/dev/profiles/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - cert-manager.yaml
  - cnpg.yaml

Removing a profile from the list prunes its resources on the next reconcile. Heavy profiles (observability, messaging, keycloak, elastic, e-commerce) should not run more than two at a time on a 16 GB laptop.

Profiles that list secrets read credentials from the external dev Vault (VAULT=1 make dev-services-up); only observability is fully self-contained.

external-dns needs the lab’s BIND server reachable at EXTERNAL_DNS_SERVER (10.0.0.11) and the TSIG key under bind9/ddns in the dev Vault. The elastic profile stores data on local-path and uses node.store.allow_mmap: false, so no host vm.max_map_count tuning is required.

The dev observability profile is self-contained: there is no Vault, so the Grafana instance uses the admin secret generated by the operator (grafana-admin-credentials), only the Prometheus datasource, and local-path persistence. Grafana CRDs are installed by the operator chart (install.crds: Create); prod installs them from Ansible instead.

Overlay conventions

Both environments share the same base and differ only through overlays and components:

Rules of thumb:

• Base never carries a resource that an environment must delete. If dev does not need it, it belongs in a component that only prod includes.
• Env-specific choices (replica counts, storage class, issuer, IP ranges, endpoints, retention, credentials) live in the overlay, not in base.
• Overlays add or override; they should not need $patch: delete or op: remove against base.
• Grafana dashboards live under components/monitoring and are prod-only. ServiceMonitors and PodMonitors that dev also needs live under components/metrics, so a dev profile remains usable without the observability stack.

External services

RustFS (and optional Vault) run outside the cluster in Docker Compose:

cd dev
make dev-services-up              # RustFS
VAULT=1 make dev-services-up      # RustFS + Vault
make dev-services-down            # stop, keep data
make dev-services-reset           # stop and delete data (confirmation)

The cluster reaches RustFS through rustfs.dev-external.svc.cluster.local:9091. The local dev CA is copied into POC namespaces as the rustfs-ca Secret.

POC workflow

cd dev
make dev-poc-new NAME=cnpg-barman

This creates test/cnpg-barman/ with a Namespace, ResourceQuota (requests 2 GiB, limits 3 GiB), LimitRange and a kustomization, plus a Flux Kustomization in kubernetes/clusters/dev/profiles/poc-cnpg-barman.yaml. Add that file to profiles/kustomization.yaml and commit.

Resource budget

Expected combinations:

Limitations and troubleshooting

• Longhorn is not supported in k3d. Dev uses local-path; PVCs are node-local and lost with the cluster. The longhorn/app dev overlay sets one replica and a /var/lib/longhorn data path, but Longhorn also needs iscsid/open-iscsi on the Docker host, which is not installed here, so it is kept for parity and is not validated.
• cert-manager-webhook-ionos is prod-only. Dev uses the self-signed CA issuer from the cert-manager profile; the IONOS DNS-01 webhook needs credentials that only exist in prod.
• Istio ambient is not usable on k3d. The Istio CNI plugin rewrites Cilium’s /etc/cni/net.d/05-cilium.conflist; pod sandbox creation then fails (FailedCreatePodSandBox) and the node eventually reports NotReady. The istio/app dev overlay is kept for rendering and prod parity, but there is no dev profile for Istio or Kiali. Recovery: remove the profile (Flux prunes Istio and the CNI agent restores the Cilium config), then restart the k3d node if it stays NotReady.
• The cluster is x86-only in practice; prod is hybrid ARM64/AMD64.
• The dev CA is local and untrusted. Do not reuse it outside the dev cluster.
• Flux tracks master. Branch validation must follow the flux-change procedure and be reverted afterwards.

Preparing the development environment

The following need to be installed in your local development environment

• Docker
• K3D
• kubectl
• helm

Docker

Follow official installation guide.

• Step 1. Uninstall old versions of docker

  for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done
  

• Step 2. Install packages to allow apt to use a repository over HTTPS

  sudo apt-get update
  
  sudo apt-get install \
  apt-transport-https \
  ca-certificates \
  curl \
  gnupg \
  lsb-release
  

• Step 3. Add Docker’s official GPG key

  sudo install -m 0755 -d /etc/apt/keyrings
  curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.asc
  sudo chmod a+r /etc/apt/keyrings/docker.asc
  

• Step 4: Add x86_64 repository

  echo \
    "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
    $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  

• Step 5: Install Docker Engine

  sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
  

• Step 6: Enable Docker management with a non-privileged user

  • Create docker group

    sudo groupadd docker
    

  • Add user to docker group

    sudo usermod -aG docker $USER
    

• Step 7: Configure Docker to start on boot

  sudo systemctl enable docker.service
  sudo systemctl enable containerd.service
  

• Step 8: Configure docker daemon.

  • Edit file /etc/docker/daemon.json

    Set storage driver to overlay2 and to use systemd for the management of the container’s cgroups. Optionally default directory for storing images/containers can be changed to a different disk partition (example /data). Documentation about the possible options can be found here

    {
        "exec-opts": ["native.cgroupdriver=systemd"],
        "log-driver": "json-file",
        "log-opts": {
        "max-size": "100m"
        },
        "storage-driver": "overlay2",
        "data-root": "/data/docker"  
    }
    

  • Restart docker

    sudo systemctl restart docker
    

K3D

Follow official K3D installation guide.

• Step 1. Uninstall old versions of docker

  wget -q -O - https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash
  

• Step 3. Check k3d version installed

  $ k3d version
  k3d version v5.8.3
  k3s version v1.37.1-k3s1 (default)
  

Kubectl

Follow official kubectl installation guide

• Step 1: Download latest stable version

  curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
  

• Step 2: Install kubectl in /usr/local/bin

  sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
  

• Step 3: Check version installed

  $ kubectl version --client --output=yaml
  clientVersion:
    buildDate: "2024-06-11T20:29:44Z"
    compiler: gc
    gitCommit: 39683505b630ff2121012f3c5b16215a1449d5ed
    gitTreeState: clean
    gitVersion: v1.30.2
    goVersion: go1.22.4
    major: "1"
    minor: "30"
    platform: linux/amd64
  kustomizeVersion: v5.0.4-0.20230601165947-6ce0bf390ce3
  

Cluster contexts

One kubeconfig can describe both clusters; they are distinguished by context name:

k3s_install.yml writes the production entry to $HOME/.kube/pi-cluster.yaml and merges it into $HOME/.kube/config, so reinstalling the lab never deletes the dev context. Pass --context rather than relying on the current context:

kubectl --context picluster-prod get nodes
kubectl --context k3d-picluster get pods -A

Running any dev/ target switches the current context to k3d-picluster.

flux takes the same global flags (--context, --kubeconfig, before or after the subcommand) and honours KUBECONFIG, so scripts can target a cluster without touching the current context:

flux --context picluster-prod get kustomizations -A
flux --context k3d-picluster reconcile kustomization flux-system --with-source

Helm

• Step 1: Download helm install script

  curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
  $ chmod 700 get_helm.sh
  $ ./get_helm.sh
  

• Step 2: Give it executable permissions

  chmod 700 get_helm.sh
  

• Step 3: Install helm

  ./get_helm.sh
  

• Step 4: Check version

  $ helm version
  version.BuildInfo{Version:"v3.15.3", GitCommit:"3bb50bbbdd9c946ba9989fbe4fb4104766302a64", GitTreeState:"clean", GoVersion:"go1.22.5"}
  

Creating dockerized k3s cluster

k3s cluster can be created using K3D. The cluster will be configured with same options as production environment:

• CNI cilium is used, instead of default Flannel
• All K3s add-ons are not installed (traefik, helmcontroller, localpath, coredns, etc.)
• kube-proxy is disabled (Cilium Kube-proxy replacement feature will be used)
• Cilium L2 LB announcement will be used

Prerequisites

Enable kernel modules

Since cilium uses IPtables to write routes to kernel, we need to enable additional kernel modules

sudo modprobe -v iptable_filter  
sudo modprobe -v ip_tables  
sudo modprobe -v iptable_mangle  
sudo modprobe -v iptable_raw  
sudo modprobe -v iptable_nat  
sudo modprobe -v xt_socket

Add this configuration in kernel modules configuration file to persist the modules after restart

echo "iptable_filter
ip_tables
iptable_mangle
iptable_raw
iptable_nat
xt_socket" | sudo tee /etc/modules-load.d/modules.conf

Ensure sysctl parameters are enabled

echo "net.core.devconf_inherit_init_net=1
net.netfilter.nf_conntrack_max=196608
net.ipv4.conf.all.forwarding = 1 | sudo tee /etc/sysctl.d/01-sysctl.conf > /dev/null

sudo sysctl -p

Creating docker network

A specific docker network, 172.30.0.0/16, will be created for the k3d dev environment

docker network create \
      --driver bridge \
      --subnet "172.30.0.0/16" \
      --gateway "172.30.0.1" \
      --ip-range "172.30.0.0/17" \
      "picluster"

Creating K3D cluster

• Step 1. Create k3d-config.yaml file

  In the repository this file is generated from dev/k3d-cluster/k3d-cluster.yaml.tmpl by dev/create_cluster.sh (make -C dev create-k3d). The rendered example is:

  apiVersion: k3d.io/v1alpha5
  kind: Simple
  metadata:
    name: picluster
  servers: 1
  # Default dev topology is single-node. Use K3D_AGENTS=1 (or 2) for
  # scheduling/failover POCs; multi-node partitions the same laptop memory.
  agents: 0
  kubeAPI: # same as `--api-port myhost.my.domain:6445` (where the name would resolve to 127.0.0.1)
    hostIP: 127.172.30.1 # where the Kubernetes API will be listening on
    hostPort: "6443"
  # Setting version of k3s
  # Same k3s minor as prod; override with K3S_IMAGE for a POC.
  image: rancher/k3s:v1.37.1-k3s1
  # Setting docker network to be used
  network: picluster
  options:
    # Disable Load balancer
    k3d:
      disableLoadbalancer: true
    # Extra arguments for k3s
    k3s:
      extraArgs:
        - arg: --tls-san=127.0.0.1
          nodeFilters:
            - server:*
        # Disable helmcontroller
        - arg: --disable-helm-controller
          nodeFilters:
            - server:*
        # Disable coreDNS
        - arg: --disable=coredns
          nodeFilters:
            - server:*
        # Disable traefik
        - arg: --disable=traefik
          nodeFilters:
            - server:*
        # Disable servicelb
        - arg: --disable=servicelb
          nodeFilters:
            - server:*
        # local-path is left enabled and is the default StorageClass.
        # Disable metric server
        - arg: --disable=metrics-server
          nodeFilters:
            - server:*
        # Disable network-policy
        - arg: --disable-network-policy
          nodeFilters:
          - server:*
        # Disable kube-proxy
        - arg: --disable-kube-proxy
          nodeFilters:
          - server:*
        # Disable flannel-backend
        - arg: --flannel-backend=none
          nodeFilters:
          - server:*
        # Adding cluster PODs and Services CIDRs
        - arg: --cluster-cidr=10.42.0.0/16
          nodeFilters:
            - server:*
        - arg: --service-cidr=10.43.0.0/16
          nodeFilters:
            - server:*
        # Exposing metrics
        - arg: --kube-controller-manager-arg=bind-address=0.0.0.0
          nodeFilters:
            - server:*
        - arg: --kube-scheduler-arg=bind-address=0.0.0.0
          nodeFilters:
            - server:*
    kubeconfig:
      updateDefaultKubeconfig: true
      switchCurrentContext: true  
  

  • By default this creates a single-node cluster (1 control plane node and 0 workers) connected to the docker network created before (picluster):

    • k3d-picluster-server-0, with IP 172.30.0.2

    Set K3D_AGENTS=1 (or 2) in dev/create_cluster.sh to add workers; each agent gets the next 172.30.0.x address.

  • k3s version is defined through the corresponding docker image label (image: rancher/k3s:v1.37.1-k3s1), matching production.

  • Following k3s services are disabled: kube-proxy, traefik, servicelb, flannel, network-policy and metrics-server (Flux installs metrics-server). local-path is left enabled and is the default StorageClass.

  • k3d default load balancer is also disabled (k3d.disableLoadbalancer: true)

• Step 2: Create K3d cluster

  K3D_FIX_MOUNTS=1 K3D_FIX_DNS=0 k3d cluster create -c k3d-config.yaml
  

  In the repository, make -C dev create-k3d renders the template and runs the equivalent command. make -C dev dev-up also installs Cilium, CoreDNS and Flux afterwards.

  Note:

  K3D_FIX_MOUNTS environment variable need to be set before executing k3d command. That makes / in the node containers rshared mounts fixing issues with Cilium installation. See details in this these k3d issues: #1268 and #479.

  K3D_FIX_DNS environment variable need to be set also, so K3d DNS fix that is executed by default is disabled.

  Purpose of this DNS fix is to update defautl behaviour of DNS in docker. By default, containers use the Docker daemon’s embedded DNS resolver, which listens on the default DNS server IP address 127.0.0.11. This DNS server handles DNS queries from containers and forwards them to the appropriate DNS server for resolution.

  K3D_FIX_DNS variable makes k3d to execute a fixing script that modifies /etc/resolv.conf in the container to point to host IP address, so host stub DNS resolver can be used.

  However, DNS fix (K3D_FIX_DNS=1) breaks DNS resolution for external domain names (Internet)”: See details in open k3d issue: #1515

• Step 3: Check cluster is running

  kubectl get nodes
  

  Note: The node stays NotReady until a CNI is installed. This is expected because flannel and kube-proxy are disabled; continue with the Cilium installation.

Installing Cilium CNI

In the automated workflow, dev/bootstrap.sh installs the same chart and Flux adopts the release. The manual steps below are the equivalent for debugging or for building the environment by hand.

Installation using Helm (Release 3):

• Step 1: Add Cilium Helm repository:

    helm repo add cilium https://helm.cilium.io/
  

• Step2: Fetch the latest charts from the repository:

    helm repo update
  

• Step 4: Create helm values file cilium-values.yml

  # Cilium operator config
  operator:
    replicas: 1
    # Roll out cilium-operator pods automatically when configmap is updated.
    rollOutPods: true
  
    # Install operator on control-plane node
    nodeSelector:
      node-role.kubernetes.io/control-plane: "true"
  
  # Roll out cilium agent pods automatically when ConfigMap is updated.
  rollOutCiliumPods: true
  
  # K8s API service
  # K3s API server inside the k3d node
  k8sServiceHost: 127.0.0.1
  k8sServicePort: 6443
  
  # Replace Kube-proxy
  kubeProxyReplacement: true
  kubeProxyReplacementHealthzBindAddr: 0.0.0.0:10256
  
  # Configure IP Address Management mode.
  # https://docs.cilium.io/en/stable/network/concepts/ipam/
  ipam:
    operator:
      clusterPoolIPv4PodCIDRList: "10.42.0.0/16"
  
  # Configure L2 announcements (LB-IPAM configuration)
  l2announcements:
    enabled: true
  externalIPs:
    enabled: true
  
  # Increase the k8s api client rate limit to avoid being limited due to increased API usage 
  k8sClientRateLimit:
    qps: 50
    burst: 200
    
  # Istio configuration
  # https://docs.cilium.io/en/latest/network/servicemesh/istio/
  # Disable socket lb for non-root ns. This is used to enable Istio routing rules
  socketLB:
    hostNamespaceOnly: true
  # Istio uses a CNI plugin to implement functionality for both sidecar and ambient modes. 
  # To ensure that Cilium does not interfere with other CNI plugins on the node,
  cni:
    exclusive: false
  

• Step 5: Install Cilium in kube-system namespace

  helm upgrade --install cilium cilium/cilium \
    --version 1.20.4 \
    --namespace kube-system \
    -f cilium-values.yaml \
    --wait
  

• Step 1: Configure Cilium LB-IPAM

  Create the following manifest file: cilium-config.yaml

    ---
    apiVersion: "cilium.io/v2alpha1"
    kind: CiliumLoadBalancerIPPool
    metadata:
      name: "first-pool"
      namespace: kube-system
    spec:
      blocks:
        - start: "172.30.200.1"
          stop: "172.30.200.254"
  
    ---
    apiVersion: cilium.io/v2alpha1
    kind: CiliumL2AnnouncementPolicy
    metadata:
      name: default-l2-announcement-policy
      namespace: kube-system
    spec:
      externalIPs: true
      loadBalancerIPs: true
  

  Apply the manifest file

   kubectl apply -f cilium-config.yaml
  

• Step 6: Confirm that the deployment succeeded, run:

  kubectl -n kube-system get pod
  

References

• Testing cilium with k3d and kind
• Harnessing the Power of Cilium: A Guide to BGP Integration with Gateway API on IPv4
• Integrating Cilium with Gateway API, IPv6, and BGP for Advanced Networking Solutions
• Understanding Docker DNS
• k3d documentation
• FluxCD documentation
• RustFS Docker installation
• dev/README.md — automated workflow quickstart
• dev/external-services/README.md — RustFS/Vault details