Pi Cluster Documentation
Pi Cluster Documentation

S3 Backup Backend (Minio)

Minio can be deployed as a Kuberentes service or as stand-alone in bare-metal environment. Since I want to use Minio Server for backing-up/restoring the cluster itself, I will go with a bare-metal installation, considering Minio as an external service in Kubernetes.

Official documentation can be used for installing stand-alone Minio Server in bare-metal environment.

For installing Minio S3 storage server, a VM (Ubuntu OS) hosted in Public Cloud or any linux server/VM that is not not part of the cluster can be used.

Minio installation and configuration tasks can be automated using Ansible role: ricsanfre.minio. This role can be used to install Minio Server and Minio Client and automate the creation of S3 buckets, users and ACLs for securing the access.

Minio installation (baremetal server)

Official documentation can be used for installing stand-alone Minio Server in bare-metal environment.

  • Step 1. Create minio’s UNIX user/group

    sudo groupadd minio
    sudo useradd minio -g minio
    
  • Step 2. Create minio’s S3 storage directory

    sudo mkdir /storage/minio
    chown -R minio:minio /storage/minio
    chmod -R 750 /storage/minio
    
  • Step 3. Create minio’s config directories

    sudo mkdir -p /etc/minio
    sudo mkdir -p /etc/minio/ssl
    sudo mkdir -p /etc/minio/policy
    chown -R minio:minio /etc/minio
    chmod -R 750 /etc/minio
    
  • Step 4. Download server binary (minio) and minio client (mc) and copy them to /usr/local/bin

    wget https://dl.min.io/server/minio/release/linux-${ARCH}/minio
    wget https://dl.minio.io/client/mc/release/linux-${ARCH}/mc
    chmod +x minio
    chmod +x mc
    sudo mv minio /usr/local/bin/minio
    sudo mv mc /usr/local/bin/mc
    

    where ${ARCH} is amd64 or arm64 depending on the server architecture.

  • Step 5: Create minio Config file /etc/minio/minio.conf

    This file contains environment variables that will be used by minio server.

    # Minio local volumes.
    MINIO_VOLUMES="/storage/minio"
    
    # Minio options.
    MINIO_OPTS="--address :9091 --console-address :9092 --certs-dir /etc/minio/ssl"
    
    # Access Key of the server.
    MINIO_ROOT_USER="<admin_user>"
    # Secret key of the server.
    MINIO_ROOT_PASSWORD="<admin_user_passwd>"
    # Minio server region
    MINIO_SITE_REGION="eu-west-1"
    

    Minio is configured with the following parameters:

    • Minio Server API Port 9091 (MINIO_OPTS=”–address :9091”)
    • Minio Console Port: 9092 (MINIO_OPTS=”–console-address :9092”)
    • Minio Storage data dir (MINIO_VOLUMES): /storage/minio
    • Minio Site Region (MINIO_SITE_REGION): eu-west-1
  • Step 6. Create systemd minio service file /etc/systemd/system/minio.service

    [Unit]
    Description=MinIO
    Documentation=https://docs.min.io
    Wants=network-online.target
    After=network-online.target
    AssertFileIsExecutable=/usr/local/bin/minio
    
    [Service]
    WorkingDirectory=/usr/local/
    
    User=minio
    Group=minio
    ProtectProc=invisible
    
    EnvironmentFile=/etc/minio/minio.conf
    ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/minio/minio.conf\"; exit 1; fi"
    
    ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES
    
    # Let systemd restart this service always
    Restart=always
    
    # Specifies the maximum file descriptor number that can be opened by this process
    LimitNOFILE=65536
    
    # Specifies the maximum number of threads this process can create
    TasksMax=infinity
    
    # Disable timeout logic and wait until process is stopped
    TimeoutStopSec=infinity
    SendSIGKILL=no
    
    [Install]
    WantedBy=multi-user.target
    

    This service start minio server using minio UNIX group, loading environment variables located in /etc/minio/minio.conf and executing the following startup command:

    /usr/local/minio server $MINIO_OPTS $MINIO_VOLUMES
    
  • Step 7. Enable minio systemd service

    sudo systemctl enable minio.service
    
  • Step 8. Start minio service

    sudo systemctl start minio.service
    
  • Step 9: Check service is online and functional:

    sudo systemctl status minio.service
    journalctl -f -u minio.service
    

Enable TLS

MinIO enables Transport Layer Security (TLS) 1.2+ automatically upon detecting a valid x.509 certificate (.crt) and private key (.key) in the MinIO cert directory

Minio cert directory

By default minio certs directory is ${HOME}/.minio/certs directory. Directory can be changed starting minio server with --cert-dir parameter

minio server --certs-dir /opt/minio/certs ...

Create Minio TLS certificate

Trusted Certificate with Let’s Encrypt

In case you have your own domain, a valid TLS certificate signed by Letsencrypt can be obtained for Minio server, using Certbot.

See certbot installation instructions and how to issue certificates in “PiCluster - TLS Certificates (Certbot)”.

Private PKI

If generating public trusted TLS certificate is not possible, selfsigned certificates with a custom CA can be used instead.

Follow this procedure for creating a self-signed certificate for Minio Server

  • Step 1. Create Root CA

    • Create Root CA Key

      openssl genrsa -out rootCA.key 4096
      
    • Create and self sign the Root Certificate

      openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.crt
      
  • Step 2. Create the signing request (csr)

    • Create key

      openssl genrsa -out private.key 4096
      
    • Create a file named openssl.conf with the content below. Set IP.1 and/or DNS.1 to point to the correct IP/DNS addresses:

      [req]
      distinguished_name = req_distinguished_name
      x509_extensions = v3_req
      prompt = no
      
      [req_distinguished_name]
      C = ES
      ST = Madrid
      L = Somewhere
      O = MyOrg
      OU = MyOU
      CN = MyServerName
      
      [v3_req]
      subjectAltName = @alt_names
      
      [alt_names]
      IP.1 = 127.0.0.1
      DNS.1 = myserver.mydomain.com
      

      Run openssl by specifying the configuration file and enter a passphrase if prompted:

      openssl req -new -x509 -nodes -days 730 -key private.key -out public.csr -config openssl.conf
      
    • Verify the csr’s content

      openssl req -in public.csr -noout -text
      
    • Generate the certificate using the mydomain csr and key along with the CA Root key

      openssl x509 -req -in public.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out public.crt -days 500 -sha256
      

Install Minio TLS certificate

Once the certificate is created, public certificate and private key need to be installed in Minio server following this procedure:

  1. Update Minio config file /etc/minio/minio.conf

     # Minio options.
     MINIO_OPTS="--address :9091 --console-address :9092 --certs-dir /etc/minio/ssl"
    

    Adding --certs-dir option Minio will look for valid X.509 certificates in /etc/minio/ssl.

  2. Copy public certificate minio.crt as /etc/minio/ssl/public.crt

     sudo cp minio.crt /etc/minio/ssl/public.crt
     sudo chown minio:minio /etc/minio/ssl/public.crt
    
  3. Copy private key minio.key as /etc/minio/ssl/private.key

     cp minio.key /etc/minio/ssl/private.key
     sudo chown minio:minio /etc/minio/ssl/private.key
    
  4. Restart minio server.

     sudo systemctl restart minio.service
    

Minio Configuration

Install Minio client

Minio client can be installed in any server to perform management operations remotely

  • Step 1: Download mc binary

    cd tmp
    wget  https://dl.min.io/client/mc/release/linux-${ARCH}$/mc
    

    Where:

    • ${ARCH} is amd64 or amr64 depending on the architecture of the linux host
  • Step 2: Move binary to /usr/local/bin

    sudo mv /tmp/mc /usr/local/bin
    chmod +x /user/local/bin/mc
    
  • Step 4: Configure connection alias to minio server.

    mc alias set ${MINIO_ALIAS} ${MINIO_URL} ${ACCESS_KEY} ${SECRET_KEY}
    

    where

    • ${MINIO_ALIAS} is a connection alias assigned to the S3 server
    • ${MINIO_URL}: Containing URL of the service (i.e.: “https://${S3_FQDN}:9091”, “https://s3.mydomain.com:9091”)
    • ${ACCESS_KEY}: Contains Minio root user configured during installation
    • ${SECRET_KEY} : Contains minio root user password configured during installation
  • Step 5: Test client

    mc admin info ${MINIO_ALIAS}
    

Buckets

Buckets can be created using Minio’s CLI (mc)

mc mb ${MINIO_ALIAS}/${MINIO_BUCKET}

Where:

  • ${MINIO_ALIAS} is the mc’s alias connection to Minio Server created during Minio Client configuration
  • ${MINIO_BUCKET} is the name of the bucket to be created

Users and ACLs

Users can be created usinng Minio’s CLI

mc admin user add ${MINIO_ALIAS} ${USER_NAME} ${USER_PASSWORD}

Where:

  • ${MINIO_ALIAS} is the mc’s alias connection to Minio Server created during Minio Client configuration
  • ${MINIO_USER} is the name of the user to be created
  • ${MINIO_PASSWORD} is the password assigned to the user

Access policies to the different buckets can be assigned to the different users using the command:

mc admin policy add ${MINIO_ALIAS} ${USER_NAME} user_policy.json

Where user_policy.json, contains AWS access policies definition like:

{
"Version": "2012-10-17",
"Statement": [
  {
      "Effect": "Allow",
      "Action": [
          "s3:DeleteObject",
          "s3:GetObject",
          "s3:ListBucket",
          "s3:PutObject"
      ],
      "Resource": [
          "arn:aws:s3:::bucket_name",
          "arn:aws:s3:::bucket_name/*"
      ]
  }  
]
}

This policy grants read-write access to bucket_name. For each user a different json need to be created, granting access to dedicated bucket.

Observability

Metrics

Minio exposes Prometheus-based metrics

For details see Minio’s documentation: “Collect MinIO Metrics Using Prometheus”.

Prometheus Integration

By default, MinIO requires authentication to scrape the metrics endpoints.

To generate the needed bearer tokens, use mc admin prometheus generate.

Authentication can also be disabled for metrics endpoint by setting MINIO_PROMETHEUS_AUTH_TYPE to public.

  1. Generate bearer token to be able to access to Minio Metrics

     mc admin prometheus generate <alias>
    

    Output is something like this:

     scrape_configs:
     - job_name: minio-job
     bearer_token: eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjQ3OTQ4Mjg4MTcsImlzcyI6InByb21ldGhldXMiLCJzdWIiOiJtaW5pb2FkbWluIn0.mPFKnj3p-sPflnvdrtrWawSZn3jTQUVw7VGxdBoEseZ3UvuAcbEKcT7tMtfAAqTjZ-dMzQEe1z2iBdbdqufgrA
     metrics_path: /minio/v2/metrics/cluster
     scheme: https
     static_configs:
     - targets: ['127.0.0.1:9091']
    

    Where:

    • bearer_token is the token to be used by Prometheus for authentication purposes
    • metrics_path is the path to scrape the metrics on Minio server (TCP port 9091)
  2. Generate Scrape configurations3 mc admin prometheus generate command can be used to generate the scrape configuration to be used by Prometheus.

    • Generate cluster metrics scrapping configuration

      mc admin prometheus generate <alias>
      

      Output is something like:

      scrape_configs:
      - job_name: minio-job
        bearer_token: eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwcm9tZXRoZXVzIiwic3ViIjoicm9vdCIsImV4cCI6NDkwMDU2MzU1OH0.X06rEpJjwe-C9KHKKu08mZU3q5ZbXF9TKQtpgmnV93aBgJtMF5co-hwzcxymYdaYRTxxydMWxLTVwlr8rdLXZw
        metrics_path: /minio/v2/metrics/cluster
        scheme: https
        static_configs:
        - targets: ['127.0.0.1:9091']
      
    • Generate node metrics scrapping configuration

      mc admin prometheus generate myminio node
      

      Output like:

      scrape_configs:
      - job_name: minio-job-node
        bearer_token: eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwcm9tZXRoZXVzIiwic3ViIjoicm9vdCIsImV4cCI6NDkwMDU2MzU3Nn0.MSptk1y2vV-Ek_b3y9hU7I9WRwSCngCVq1gR_FuTXAHSSg2Pxd4p0TPlTY-_Z2SZEABwaati5Eaila3-Zi9iKg
        metrics_path: /minio/v2/metrics/node
        scheme: https
        static_configs:
        - targets: ['127.0.0.1:9091']
      
    • Generate bucket metrics scrapping configuration

      mc admin prometheus generate myminio bucket
      

      Output like:

      scrape_configs:
      - job_name: minio-job-bucket
        bearer_token: eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwcm9tZXRoZXVzIiwic3ViIjoicm9vdCIsImV4cCI6NDkwMDU2MzYxM30.7czrNI5DYC5gJqVlF3x77Sw0Gln1iJCUuzBBcA72kBL88QEjlBOuDqtbVaB8osoniQzRNDK8jYH0FwAAHKY9zw
        metrics_path: /minio/v2/metrics/bucket
        scheme: https
        static_configs:
        - targets: ['127.0.0.1:9091']
      
    • Generate bucket metrics scrapping configuration

      mc admin prometheus generate myminio resource
      

      Output like:

      scrape_configs:
      - job_name: minio-job-resource
        bearer_token: eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwcm9tZXRoZXVzIiwic3ViIjoicm9vdCIsImV4cCI6NDkwMDU2MzYzMX0.MCuM3w00Q25b4bWprij0OaMs53wdv9_IPSp5yKVW0IEQlpkFeaPrVTh0Vr5w3pYH2EquToXmWE594EUaRIogsw
        metrics_path: /minio/v2/metrics/resource
        scheme: https
        static_configs:
        - targets: ['127.0.0.1:9091']
      

Monitoring from Kube-Prometheus-Stack

In case Prometheus server is deployed in Kuberentes cluster using kube-prometheus-stack (i.e Prometheus Operator), Prometheus Operator CRD ScrapeConfig resource can be used to automatically add configuration for scrapping metrics from baremetal Minio.

  • Create Kubernetes secret containing bearer token generated before

    apiVersion: v1
    kind: Secret
    type: Opaque
    metadata:
      name: minio-monitor-token
    data:
      token: < minio_bearer_token | b64encode >
    
  • Create Prometheus Operator ScrapeConfig resources

    ---
    apiVersion: monitoring.coreos.com/v1alpha1
    kind: ScrapeConfig
    metadata:
      name: minio-job
    spec:
      jobName: minio-ext
      authorization:
        credentials:
          name: minio-monitor-token
          key: token
      metricsPath: /minio/v2/metrics/cluster
      scheme: HTTPS
      staticConfigs:
      - targets:
        - ${S3_BACKUP_SERVER}:9091
    ---
    apiVersion: monitoring.coreos.com/v1alpha1
    kind: ScrapeConfig
    metadata:
      name: minio-job-node
    spec:
      jobName: minio-ext
      authorization:
        credentials:
          name: minio-monitor-token
          key: token
      metricsPath: /minio/v2/metrics/node
      scheme: HTTPS
      staticConfigs:
      - targets:
        - ${S3_BACKUP_SERVER}:9091
    ---
    apiVersion: monitoring.coreos.com/v1alpha1
    kind: ScrapeConfig
    metadata:
      name: minio-job-bucket
    spec:
      jobName: minio-ext
      authorization:
        credentials:
          name: minio-monitor-token
          key: token
      metricsPath: /minio/v2/metrics/bucket
      scheme: HTTPS
      staticConfigs:
      - targets:
        - ${S3_BACKUP_SERVER}:9091
    ---
    apiVersion: monitoring.coreos.com/v1alpha1
    kind: ScrapeConfig
    metadata:
      name: minio-job-resource
    spec:
      jobName: minio-ext
      authorization:
        credentials:
          name: minio-monitor-token
          key: token
      metricsPath: /minio/v2/metrics/resource
      scheme: HTTPS
      staticConfigs:
      - targets:
        - ${S3_BACKUP_SERVER}:9091
    

    Where ${S3_BACKUP_SERVER} should be replaced by DNS or IP address of the backup server.

Grafana Dashboards

MinIO provides Grafana Dashboards to display metrics collected by Prometheus.

There are 3 Dashboards available:

Dashboard can be automatically added using Grafana’s dashboard providers configuration. See further details in “PiCluster - Observability Visualization (Grafana): Automating installation of community dasbhoards

Add following configuration to Grafana’s helm chart values file, so a MinIO’s dashboard provider can be created and dashboards can be automatically downloaded from GitHub repository

dashboardProviders:
  dashboardproviders.yaml:
    apiVersion: 1
    providers:
      - name: minio
        orgId: 1
        folder: Minuo
        type: file
        disableDeletion: false
        editable: true
        options:
          path: /var/lib/grafana/dashboards/minio-folder
# Dashboards
dashboards:
  minio:
    minio-server:
      # https://grafana.com/grafana/dashboards/13502-minio-dashboard/
      # renovate: depName="MinIO Dashboard"
      gnetId: 13502
      revision: 26
      datasource:
        - { name: DS_PROMETHEUS, value: Prometheus }
    minio-bucket:
      # https://grafana.com/grafana/dashboards/19237-minio-bucket-dashboard/
      # renovate: depName="MinIO Dashboard"
      gnetId: 19237
      revision: 2
      datasource:
        - { name: DS_PROMETHEUS, value: Prometheus }
    minio-node:
      url: https://raw.githubusercontent.com/minio/minio/master/docs/metrics/prometheus/grafana/node/minio-node.json
      datasource: Prometheus
    minio-replication:
      # https://grafana.com/grafana/dashboards/15305-minio-replication-dashboard/
      # renovate: depName="MinIO Dashboard"
      gnetId: 15305
      revision: 5
      datasource:
        - { name: DS_PROMETHEUS, value: Prometheus }


Last Update: Jun 24, 2025

Comments: