Minio S3 Object Storage Service

Minio will be deployed as a Kuberentes service providing Object Store S3-compatile backend for other Kubernetes Services (Loki, Tempo, Mimir, etc. )

Official Minio Kubernetes installation documentation uses Minio Operator to deploy and configure a multi-tenant S3 cloud service.

Instead of using Minio Operator, Vanilla Minio helm chart will be used. Not need to support multi-tenant installations and Vanilla Minio helm chart supports also the automatic creation of buckets, policies and users. Minio Operator creation does not automate this process.

Minio installation

Installation using Helm (Release 3):

• Step 1: Add the Minio Helm repository:

  helm repo add minio https://charts.min.io/
  

• Step2: Fetch the latest charts from the repository:

  helm repo update
  

• Step 3: Create namespace

  kubectl create namespace minio
  

• Step 3: Create Minio secret

  The following secret need to be created, containing Minio’s root user and password, and keys from others users that are going to be provisioned automatically when installing the helm chart (loki, tempo):

  apiVersion: v1
  kind: Secret
  metadata:
    name: minio-secret
    namespace: minio
  type: Opaque
  data:
    rootUser: < minio_root_user | b64encode >
    rootPassword: < minio_root_key | b64encode >
    lokiPassword: < minio_loki_key | b64encode >
    tempoPassword: < minio_tempo_key | b64encode >
  

• Step 4: Create file minio-values.yml

  # Get root user/password from secret
  existingSecret: minio-secret
  
  # Number of drives attached to a node
  drivesPerNode: 1
  # Number of MinIO containers running
  replicas: 3
  # Number of expanded MinIO clusters
  pools: 1
  # Persistence
  persistence:
    enabled: true
    storageClass: "longhorn"
    accessMode: ReadWriteOnce
    size: 10Gi
  
  # Resource request
  resources:
    requests:
      memory: 1Gi
  
  # Minio Buckets
  buckets:
    - name: k3s-loki
      policy: none
    - name: k3s-tempo
      policy: none
  
  # Minio Policies
  policies:
    - name: loki
      statements:
        - resources:
            - 'arn:aws:s3:::k3s-loki'
            - 'arn:aws:s3:::k3s-loki/*'
          actions:
            - "s3:DeleteObject"
            - "s3:GetObject"
            - "s3:ListBucket"
            - "s3:PutObject"
    - name: tempo
      statements:
        - resources:
            - 'arn:aws:s3:::k3s-tempo'
            - 'arn:aws:s3:::k3s-tempo/*'
          actions:
            - "s3:DeleteObject"
            - "s3:GetObject"
            - "s3:ListBucket"
            - "s3:PutObject"
            - "s3:GetObjectTagging"
            - "s3:PutObjectTagging"
  # Minio Users
  users:
    - accessKey: loki
      existingSecret: minio-secret
      existingSecretKey: lokiPassword
      policy: loki
    - accessKey: tempo
      existingSecret: minio-secret
      existingSecretKey: tempoPassword
      policy: tempo
  

  With this configuration:

  • Minio cluster of 3 nodes (replicas) is created with 1 drive per node (drivesPerNode) of 10Gb (persistence)

  • Root user and passwork is obtained from the secret created in Step 3 (existingSecret).

  • Memory resources for each replica is set to 1GB (resources.requests.memory). Default config is 16GB which is not possible in a Raspberry Pi.

  • Buckets (buckets), users (users) and policies (policies) are created for Loki and Tempo

• Step 5: Install Minio in minio namespace

  helm install minio minio -f minio-values.yml --namespace minio
  

• Step 6: Check status of Loki pods

  kubectl get pods -l app.kubernetes.io/name=minio -n minio
  

Configuring Ingress

Create a Ingress rule to make Minio console and service API available through the Ingress Controller (Traefik) using a specific URLs (minio.picluster.ricsanfre.com and s3.picluster.ricsanfre.com), mapped by DNS to Traefik Load Balancer external IP.

Minio backend is deployed in insecure mode (TLS is not activated) and thus Ingress resource will be configured to enable HTTPS (Traefik TLS end-point). The following configuration assumes that, Traefik is already configured for redirecting all HTTP traffic to HTTPS.

• Step 1. Create a manifest file minio_ingress.yml for providing access to Minio API.

  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: minio-ingress
    namespace: minio
    annotations:
      # HTTPS as entry point
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
      # Enable TLS
      traefik.ingress.kubernetes.io/router.tls: "true"
      # Enable cert-manager to create automatically the SSL certificate and store in Secret
      cert-manager.io/cluster-issuer: ca-issuer
      cert-manager.io/common-name: s3.picluster.ricsanfre.com
  spec:
    tls:
      - hosts:
          - s3.picluster.ricsanfre.com
        secretName: minio-tls
    rules:
      - host: s3.picluster.ricsanfre.com
        http:
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: minio
                  port:
                    number: 9000
  

• Step 2. Create a manifest file minio_console_ingress.yml for providing access to Minio Console.

  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: minio-console-ingress
    namespace: minio
    annotations:
      # HTTPS as entry point
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
      # Enable TLS
      traefik.ingress.kubernetes.io/router.tls: "true"
      # Enable cert-manager to create automatically the SSL certificate and store in Secret
      cert-manager.io/cluster-issuer: ca-issuer
      cert-manager.io/common-name: minio.picluster.ricsanfre.com
  spec:
    tls:
      - hosts:
          - minio.picluster.ricsanfre.com
        secretName: minio-console-tls
    rules:
      - host: minio.picluster.ricsanfre.com
        http:
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: minio-console
                  port:
                    number: 9001