S3 buckets

Overview

This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

Credential pair

In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

Credential pair - Create

Create new EC2 credentials in OpenStack using the OpenStack-CLI:

openstack ec2 credentials create

This will give you an output in the following format:

+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                                                                          |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access     | tpvx3i0gk5rf4duomnr7davjxl517z9c                                                                                                               |
| links      | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw                                                                                                               |
| secret     | 6lifckxv1005z60csekl7qynwxwbv3re                                                                                                               |
| trust_id   | None                                                                                                                                           |
| user_id    | tpvx3i0gk5rf4duomnr7davjxl517z9c                                                                                                               |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+

Credential pair - Show

If you ever need to look the credentials up again, use the following command:

access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}

This will give you an output formatted like this:

+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                                                                          |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access     | tpvx3i0gk5rf4duomnr7davjxl517z9c                                                                                                               |
| links      | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw                                                                                                               |
| secret     | 6lifckxv1005z60csekl7qynwxwbv3re                                                                                                               |
| trust_id   | None                                                                                                                                           |
| user_id    | tpvx3i0gk5rf4duomnr7davjxl517z9c                                                                                                               |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+

Credential pair - Delete

If you need to delete your credentials, you can so like this:

access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}

When running 'delete' you should get no response apart from the status code 0.

General usage

When using the S3 technology, you have different possible cli-tools. The most popular implementations are:

• aws
• s3cmd

This page focuses on the usage of those two implementations.

General usage - Connect

General usage - Connect - AWS client

This section explains the general usage such as configuring the connection using the AWS-client.

General usage - Connect - AWS client - Installation

Install the awscli using your favorite package manager:

# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli

General usage - Connect - AWS client - Configuration

After installing the awscli package, you can configure it like so:

aws configure

The configuration helper will prompt you to enter the following information:

AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json

This will then create config files on your machine in the following locations:

• ~/.aws/config
• ~/.aws/credentials

General usage - Connect - AWS client - Cheatsheet

Short overview of available commands when using s3cmd:

Cheatsheet

Description	Command
Show available buckets	aws s3api list-buckets or aws s3 ls
Create a bucket	aws s3api create-bucket <bucket-name>
Delete a bucket	aws s3api delete-bucket --bucket <bucket-name>
Show content of a bucket	aws s3api list-objects --bucket <bucket-name>
Show all command available	aws s3api help

General usage - Connect - S3cmd

This section explains the general usage such as configuring the connection using the S3cmd-client.

General usage - Connect - S3cmd - Installation

Install the s3cmd using your favorite package manager:

# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd

General usage - Connect - S3cmd - Configuration

To configure s3cmd, create a configuration file like so:

# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg

The configuration file should include the following options:

access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000

General usage - Connect - S3cmd - Cheatsheet

Short overview of available commands when using s3cmd:

Cheatsheet

Description	Command
Show available buckets	s3cmd ls
Create a bucket	s3cmd mb s3://<bucket-name>
Delete a bucket	s3cmd rb s3://<bucket-name>
Show content of a bucket	s3cmd ls s3://<bucket-name>
Put file into bucket	s3cmd put <file> s3://<bucket-name>
Get file from bucket	s3cmd get s3://<bucket-name>/<file-name>
Delete file from bucket	s3cmd [del|rm] s3://<bucket-name>/<file-name>
Show disk usage of buckets	s3cmd du
Show all command available	s3cmd --help

Lifecycle

This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.

endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>

Lifecycle - Create bucket

This section explains how to create a new s3 bucket.

Lifecycle - Create bucket

The following commands explain how to create a normal s3-bucket.

To create a new s3-bucket using the aws-cli, use the following command:

aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}

Output:

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:

aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}

The output should then look something like this:

2024-11-26 14:46:28 test77

To create a new s3-bucket using s3cmd, use the following command:

s3cmd mb s3://${bucket_name}

Output:

s3cmd should return the following response:

Bucket 's3://test76/' created

To create a new s3-bucket using the OpenStack dashboard, follow this section.

First go to Object Store -> Containers and press + Container:

After doing that, enter a name that isn't taken yet, press Public or Not public and submit.

The s3-bucket has been created successfully if it shows up in the list of s3-buckets.

There you also have the possibility to create new folders and upload files within that s3-bucket.

Lifecycle - Create bucket with object locking

To create a s3-bucket with support for object-locking, use the following commands.

To disable versioning of a s3-bucket using the aws-cli, use the following command:

aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket

Output:

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:

aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}

The output should then look something like this:

2024-11-26 14:46:28 test77

s3cmd doesn't support object-locks.

Lifecycle - Versioning

This section explains how to enable versioning for a s3 bucket.

Lifecycle - Versioning - Get status

To get the current versioning status for a certain bucket, use the following commands.

To retrieve the status of a s3-bucket using the aws-cli, use the following command:

aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}

Output:

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:

{
    "Status": "Suspended",
    "MFADelete": "Disabled"
}

If you have versioning configured, it should look similar to this:

{  
    "Status": "Enabled",
    "MFADelete": "Disabled"
}

To retrieve the status of a s3-bucket using the s3cmd, use the following command:

s3cmd info s3://${bucket_name}

Output:

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:

s3://test77/ (bucket):
   Location:  location
   Payer:     BucketOwner
   Ownership: none
   Versioning:none
   Expiration rule: none
   Block Public Access: none
   Policy:    none
   CORS:      none
   ACL:       x: FULL_CONTROL

If you have versioning disabled, it will look similar to this:

s3://test77/ (bucket):
   Location:  location
   Payer:     BucketOwner
   Ownership: none
   Versioning:Suspended
   Expiration rule: none
   Block Public Access: none
   Policy:    none
   CORS:      none
   ACL:       x: FULL_CONTROL

If you have versioning configured, it should look similar to this:

s3://test77/ (bucket):
   Location:  location
   Payer:     BucketOwner
   Ownership: none
   Versioning:Enabled
   Expiration rule: none
   Block Public Access: none
   Policy:    none
   CORS:      none
   ACL:       x: FULL_CONTROL

Lifecycle - Versioning - Enable Versioning

To enable versioning use the following commands.

To enable versioning of a s3-bucket using the aws-cli, use the following command:

aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled

Output:

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the Get status command. The output should then look something like this:

{  
    "Status": "Enabled",
    "MFADelete": "Disabled"
}

To enable versioning of a s3-bucket using s3cmd, use the following command:

s3cmd setversioning s3://${bucket_name} enable

Output:

s3cmd should return the following response:

s3://newfreshs3/: Versioning status updated

If you then run the Get status command you will get an output like this:

s3://test77/ (bucket):
   Location:  location
   Payer:     BucketOwner
   Ownership: none
   Versioning:Enabled
   Expiration rule: none
   Block Public Access: none
   Policy:    none
   CORS:      none
   ACL:       x: FULL_CONTROL

Lifecycle - Versioning - Disable Versioning

To disable versioning use the following commands.

To disable versioning of a s3-bucket using the aws-cli, use the following command:

aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended

Output:

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the Get status command. The output should then look something like this:

{
    "Status": "Suspended",
    "MFADelete": "Disabled"
}

To enable versioning of a s3-bucket using s3cmd, use the following command:

s3cmd setversioning s3://newfreshs3 disable

Output:

s3cmd should return the following response:

s3://newfreshs3/: Versioning status updated

Lifecycle - Object lock

This section covers how to manage object locks and set retention policies.

Lifecycle - Object lock - Get retention policy

Use the following commands to retrieve the current retention policy of a s3-bucket.

To get the retention policy of a s3-bucket using the aws-cli, use the following command:

aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}

Output:

The output should look like this:

{
    "ObjectLockConfiguration": {
        "ObjectLockEnabled": "Enabled"
    }
}

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag (Create bucket with object lock enabled).

An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown

If you already have a retention policy set, then the output might look something like this:

{
    "ObjectLockConfiguration": {
        "ObjectLockEnabled": "Enabled",
        "Rule": {
            "DefaultRetention": {
                "Mode": "COMPLIANCE",
                "Days": 3
            }
        }
    }
}

Not supported by s3cmd.

Lifecycle - Object lock - Set retention policy

Use the following commands to set the retention policy of a s3-bucket.

To set the retention time of a s3-bucket to a certain amount of days, use the following command:

aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'

Output:

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the get-object-lock-configuration command. The output should then look like this:

{
    "ObjectLockConfiguration": {
        "ObjectLockEnabled": "Enabled",
        "Rule": {
            "DefaultRetention": {
                "Mode": "COMPLIANCE",
                "Days": 3
            }
        }
    }
}

Not supported by s3cmd.

Lifecycle - Policy

This section explains how to add a policy to a s3 bucket.

Lifecycle - Policy - Get status

To get the bucket policy, run the following commands.

To get the bucket policy of a s3-bucket using the aws-cli, use the following command:

aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}

Output:

If the bucket has no policy, then the output should look something like this:

An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist

if the bucket has a policy, then the output should look something like this:

{
    "Policy": "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [{\n    \"Effect\": \"Allow\",\n    \"Principal\": {\"AWS\": [\n      \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n      \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n     ]},\n    \"Action\": [\n        \"s3:ListBucket\",\n        \"s3:PutObject\",\n        \"s3:DeleteObject\",\n        \"s3:GetObject\"\n      ],\n    \"Resource\": [\n      \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n      \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n    ]\n  }]\n}\n\n"
}

To get the bucket policy of a s3-bucket using s3cmd, use the following command:

s3cmd setversioning s3://${bucket_name} enable

Output:

If the bucket has no policy, then the output will look something like this:

s3://test77/ (bucket):
   Location:  location
   Payer:     BucketOwner
   Ownership: none
   Versioning:Enabled
   Expiration rule: none
   Block Public Access: none
   Policy:    none
   CORS:      none
   ACL:       x: FULL_CONTROL

If the bucket has a policy, then the output will look something like this:

s3://secondtest/ (bucket):
   [...]
   Policy:    {
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"AWS": [
      "arn:aws:iam::x:root",
      "arn:aws:iam::x:user/testuser"
     ]},
    "Action": [
        "s3:ListBucket",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:GetObject"
      ],
    "Resource": [
      "arn:aws:s3:::x/backup-Y-bucket/*",
      "arn:aws:s3:::x/backup-Y-bucket"
    ]
  }]
}

   CORS:      [...]
   ACL:       [...]

Lifecycle - Policy - Set policy

A typical json formatted policy-file will look something like this:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"AWS": [
      "arn:aws:iam::x:root",
      "arn:aws:iam::x:user/testuser"
     ]},
    "Action": [
        "s3:ListBucket",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:GetObject"
      ],
    "Resource": [
      "arn:aws:s3:::x/backup-Y-bucket/*",
      "arn:aws:s3:::x/backup-Y-bucket"
    ]
  }]
}

The following commands are used to set the policy for a certain bucket.

To set the bucket policy of a s3-bucket using the aws-cli, use the following command:

aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json

Output:

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

To set the bucket policy of a s3-bucket using s3cmd, use the following command:

s3cmd setpolicy <policy_file>.json s3://${bucket_name}

Output:

If you ran the s3cmd command, you should get the following response:

s3://yours3bucket/: Policy updated

Lifecycle - Policy - Remove Policy

The following commands are used to remove a policy that is no longer wanted:

To remove the policy of a s3-bucket using the aws-cli, use the following command:

aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}

Output:

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

To remove the policy of a s3-bucket using s3cmd, use the following command:

s3cmd delpolicy s3://${bucket_name}

Output:

If you ran the s3cmd command, you should get the following response:

s3://newfreshs3/: Policy deleted