Data storage with MinIO

S3 (Simple Storage Service) is a cloud object storage service, first introduced in Amazon S3, offering high performance data management in a filesystem-like environment. Buckets are the root elements of MinIO and hold the configuration on storage and access-policies for all underlying data. A bucket can contain any amount of folders, subdirectories and files, fitting in the designated bucket-storage-space. Folders inside a bucket can hold own access-policies, overwriting the parent policy for this and all underlying folders and files. Files can have any format. Data can be managed directly in the User Interface or with HTTPS requests.

The UFZ currently maintains a MinIO instance as S3 storage solution.

Get MinIO access

In order to upload or download data to and from MinIO, you need an account for MinIO and access to the folders you want to manage your data with.

Get an account

There are two options for getting an account:

1. By having your name on the initial list of users in the registration process.

2. Contact the administrators via email at rdm-contact@ufz.de and state your name, the bucket(s) you need access to and what kind of access you need (upload and/or download). In this case have the bucket-owner apply for your account or add them in CC.

Configure your account

After getting an account you should create an Access Key. The Access Key allows you to manage your data via scripts, providing faster and automated data management.

Get or change access to a bucket

If you already have an account but need access to a new bucket or have an existing access-configuration modified (e.g. add upload rights), please contact the administrators at rdm-contact@ufz.de with

• your name,
• the name(s) of the bucket(s) and
• what kind of access or changes you need.

To ensure, that only authorized persons get access to a bucket, either have the bucket-owner write the mail or add them in CC.

Account configuration

After getting an account for MinIO, you need to create an Access Key:

1. Login at https://minio.ufz.de with your LDAP credentials

1. On the left click on Access Keys and then on the right Create access key.

1. Make sure to save the Access Key and Secret Key.

MinIO Client

MinIO offers data management with the command line tool MinIO Client, which is supported for Linux, macOS and Windows. The full documentation with installation instructions for all operating systems can be found here. Following is a list of some basic commands:

Add MinIO to mc configuration

bash +o history
mc alias set ALIAS HOSTNAME ACCESS_KEY SECRET_KEY
bash -o history

• ALIAS can be freely chosen as a name for the MinIO storage
• HOSTNAME must be https://minio.ufz.de
• The ACCESS_KEY and SECRET_KEY that you created in Account configuration

Copy data to or from MinIO

mc cp [--recursive ...] SOURCE [SOURCE ...] TARGET

• SOURCE can be one or multiple sources of files or directories in MinIO or local filesystem
  • MinIO: alias/bucket/path.example
  • Local: absolute or relative path to file/directory (this path will be fully copied to the MinIO structure)
• TARGET can be a destination folder or file name in MinIO or local
• If a folder with all subfolders should be moved, the option --recursive must be given

List objects on MinIO

mc ls [--recursive ...] ALIAS [ALIAS ...]

• ALIAS can be the path on MinIO or local filesystem to a file or directory

Script based data management

You can manage your data with semi-automated scripts. They access the storage via HTTPS requests. You need following scripts:

• .s3-credentials-example
• s3-upload.sh
• s3-download.sh

All scripts must stay in the same folder for them to work. You can move them together to any location.

Configuration

To upload and download data to and from the S3-Storage, adjust the connection parameters in .s3-credentials-example and save the file as .s3-credentials.

HOST="minio.ufz.de"
BUCKET="bucket"
S3_ACCESS_KEY="s3-user-name"
S3_SECRET_KEY="s3-password"

• HOST always stays "minio.ufz.de" as long as the MinIO address itself does not change.
• BUCKET can be the name of a bucket or a path to a folder inside a bucket (e.g. "bucket/folder" or "bucket/folder/subfolder").
• S3_ACCESS_KEY is the username of your Access Key.
• S3_SECRET_KEY is the password of your Access Key.

Upload

You can upload your data with s3-upload.sh. All data will be uploaded to the bucket set in .s3-credentials.

The full file-path will be created/expanded in the bucket, so make sure to call the upload from the right directory.

/path/to/s3-upload.sh [data-type] [/path/to/file/or/directory]

[data-type]
-f  Upload single file
-d  Upload folder

Examples

Upload command                                      S3-View

/path/to/s3-upload.sh -f /path/to/example.txt       # bucket/path/to/example/example.txt
/path/to/s3-upload.sh -f example.txt                # bucket/example.txt

/path/to/s3-upload.sh -d /path/to/example-folder    # bucket/path/to/example-folder
/path/to/s3-upload.sh -d example-folder             # bucket/example-folder

Download

To download data, you can use the file s3-download.sh.

If the path given does not exist locally, it will be fully created starting from the directory the script is called. If parts of the path exist, it will be expanded by the new file/folder structure.

path/to/s3-download.sh [data-type] [path/to/file/or/directory/of/bucket/or/subdirectory]

[data-type]
-f  Download single file
-d  Download folder

Example

Download single file
path/to/s3-download.sh -f example.txt

Download single file from bucket subdirectory
path/to/s3-download.sh -f example-folder/example.txt

Download all content from bucket
path/to/s3-download.sh -d ''

Download subfolder
path/to/s3-download.sh -d example-folder