On this page:

Minio S3 Object Storage Service

Minio will be deployed as a Kuberentes service providing Object Store S3-compatile backend for other Kubernetes Services (Loki, Tempo, Mimir, etc. )

Official Minio Kubernetes installation documentation uses Minio Operator to deploy and configure a multi-tenant S3 cloud service.

Instead of using Minio Operator, Vanilla Minio helm chart will be used. Not need to support multi-tenant installations and Vanilla Minio helm chart supports also the automatic creation of buckets, policies and users. Minio Operator creation does not automate this process.

Minio installation

Installation using Helm (Release 3):

  • Step 1: Add the Minio Helm repository:

    helm repo add minio https://charts.min.io/
    
  • Step2: Fetch the latest charts from the repository:

    helm repo update
    
  • Step 3: Create namespace

    kubectl create namespace minio
    
  • Step 3: Create Minio secret

    The following secret need to be created, containing Minio’s root user and password, and keys from others users that are going to be provisioned automatically when installing the helm chart (loki, tempo):

    apiVersion: v1
    kind: Secret
    metadata:
      name: minio-secret
      namespace: minio
    type: Opaque
    data:
      rootUser: < minio_root_user | b64encode >
      rootPassword: < minio_root_key | b64encode >
      lokiPassword: < minio_loki_key | b64encode >
      tempoPassword: < minio_tempo_key | b64encode >
    
  • Step 4: Create file minio-values.yml

    # Get root user/password from secret
    existingSecret: minio-secret
    
    # Number of drives attached to a node
    drivesPerNode: 1
    # Number of MinIO containers running
    replicas: 3
    # Number of expanded MinIO clusters
    pools: 1
    
    # Run minio server only on amd64 nodes
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: kubernetes.io/arch
              operator: In
              values:
              - amd64
    
    # Persistence
    persistence:
      enabled: true
      storageClass: "longhorn"
      accessMode: ReadWriteOnce
      size: 10Gi
    
    # Resource request
    resources:
      requests:
        memory: 1Gi
    
    # Service Monitor
    metrics:
      serviceMonitor:
        enabled: true
        includeNode: true
    
    # Minio Buckets
    buckets:
      - name: k3s-loki
        policy: none
      - name: k3s-tempo
        policy: none
    
    # Minio Policies
    policies:
      - name: loki
        statements:
          - resources:
              - 'arn:aws:s3:::k3s-loki'
              - 'arn:aws:s3:::k3s-loki/*'
            actions:
              - "s3:DeleteObject"
              - "s3:GetObject"
              - "s3:ListBucket"
              - "s3:PutObject"
      - name: tempo
        statements:
          - resources:
              - 'arn:aws:s3:::k3s-tempo'
              - 'arn:aws:s3:::k3s-tempo/*'
            actions:
              - "s3:DeleteObject"
              - "s3:GetObject"
              - "s3:ListBucket"
              - "s3:PutObject"
              - "s3:GetObjectTagging"
              - "s3:PutObjectTagging"
    # Minio Users
    users:
      - accessKey: loki
        existingSecret: minio-secret
        existingSecretKey: lokiPassword
        policy: loki
      - accessKey: tempo
        existingSecret: minio-secret
        existingSecretKey: tempoPassword
        policy: tempo
    
    # Ingress resource (nginx)
    ingress:
      ## Enable creation of ingress resource
      enabled: true
      ## Add ingressClassName to the Ingress
      ingressClassName: nginx
      # ingress host
      hosts:
        - s3.picluster.ricsanfre.com
      ## TLS Secret Name
      tls:
        - secretName: minio-tls
          hosts:
            - s3.picluster.ricsanfre.com
      ## Default ingress path
      path: /
      ## Ingress annotations
      annotations:
        # Linkerd configuration. Configure Service as Upstream
        nginx.ingress.kubernetes.io/service-upstream: "true"
        # Enable cert-manager to create automatically the SSL certificate and store in Secret
        # Possible Cluster-Issuer values:
        #   * 'letsencrypt-issuer' (valid TLS certificate using IONOS API)
        #   * 'ca-issuer' (CA-signed certificate, not valid)
        cert-manager.io/cluster-issuer: letsencrypt-issuer
        cert-manager.io/common-name: s3.picluster.ricsanfre.com
    
    # console Ingress (nginx)
    consoleIngress:
      ## Enable creation of ingress resource
      enabled: true
      ## Add ingressClassName to the Ingress
      ingressClassName: nginx
      # ingress host
      hosts:
        - minio.picluster.ricsanfre.com
      ## TLS Secret Name
      tls:
        - secretName: minio-console-tls
          hosts:
            - minio.picluster.ricsanfre.com
      ## Default ingress path
      path: /
      ## Ingress annotations
      annotations:
        # Linkerd configuration. Configure Service as Upstream
        nginx.ingress.kubernetes.io/service-upstream: "true"
        # Enable cert-manager to create automatically the SSL certificate and store in Secret
        # Possible Cluster-Issuer values:
        #   * 'letsencrypt-issuer' (valid TLS certificate using IONOS API)
        #   * 'ca-issuer' (CA-signed certificate, not valid)
        cert-manager.io/cluster-issuer: letsencrypt-issuer
        cert-manager.io/common-name: minio.picluster.ricsanfre.com
    
    

    With this configuration:

    • Minio cluster of 3 nodes (replicas) is created with 1 drive per node (drivesPerNode) of 10Gb (persistence)

    • Root user and passwork is obtained from the secret created in Step 3 (existingSecret).

    • Memory resources for each replica is set to 1GB (resources.requests.memory). Default config is 16GB which is not possible in a Raspberry Pi.

    • Enable creation of Prometheus ServiceMonitor object (metrics.serviceMonitor).

    • Minio PODs are deployed only on x86 nodes (affinity). Minio does not work properly when mixing nodes of different architectures. See issue #137

    • Buckets (buckets), users (users) and policies (policies) are created for Loki and Tempo

    • Ingress resource (ingress) for s3 service API available at s3.picluster.ricsanfre.com. Annotated so Cert-Manager generate the TLS certificate automatically.

    • Ingress resource (ingressConsole) for S3 console available at minio.picluster.ricsanfre.com. Annotated so Cert-Manager generate the TLS certificate automatically.

  • Step 5: Install Minio in minio namespace
    helm install minio minio -f minio-values.yml --namespace minio
    
  • Step 6: Check status of Loki pods
    kubectl get pods -l app.kubernetes.io/name=minio -n minio
    

Last Update: Jul 03, 2023

Comments: