S3 buckets
Overview
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.
Credential pair
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.
This section will guide you through the creation process in our OpenStack-based cloud.
Credential pair - Create
Create new EC2 credentials in OpenStack using the OpenStack-CLI:
openstack ec2 credentials create
This will give you an output in the following format:
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
Credential pair - Show
If you ever need to look the credentials up again, use the following command:
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
This will give you an output formatted like this:
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
Credential pair - Delete
If you need to delete your credentials, you can so like this:
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
When running 'delete' you should get no response apart from the status code 0.
General usage
When using the S3 technology, you have different possible cli-tools. The most popular implementations are:
- aws
- s3cmd
This page focuses on the usage of those two implementations.
General usage - Connect
General usage - Connect - AWS client
This section explains the general usage such as configuring the connection using the AWS-client.
General usage - Connect - AWS client - Installation
Install the awscli using your favorite package manager:
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
General usage - Connect - AWS client - Configuration
After installing the awscli package, you can configure it like so:
aws configure
The configuration helper will prompt you to enter the following information:
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
This will then create config files on your machine in the following locations:
- ~/.aws/config
- ~/.aws/credentials
General usage - Connect - AWS client - Cheatsheet
Short overview of available commands when using s3cmd:
| Description | Command |
|---|---|
| Show available buckets | aws s3api list-buckets
aws s3 ls
|
| Create a bucket | aws s3api create-bucket <bucket-name>
|
| Delete a bucket | aws s3api delete-bucket --bucket <bucket-name>
|
| Show content of a bucket | aws s3api list-objects --bucket <bucket-name>
|
| Show all command available | aws s3api help
|
General usage - Connect - S3cmd
This section explains the general usage such as configuring the connection using the S3cmd-client.
General usage - Connect - S3cmd - Installation
Install the s3cmd using your favorite package manager:
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
General usage - Connect - S3cmd - Configuration
To configure s3cmd, create a configuration file like so:
# Create file
touch ~/.s3cfg
# Edit file
vim ~/.s3cfg
The configuration file should include the following options:
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
General usage - Connect - S3cmd - Cheatsheet
Short overview of available commands when using s3cmd:
| Description | Command |
|---|---|
| Show available buckets | s3cmd ls
|
| Create a bucket | s3cmd mb s3://<bucket-name>
|
| Delete a bucket | s3cmd rb s3://<bucket-name>
|
| Show content of a bucket | s3cmd ls s3://<bucket-name>
|
| Put file into bucket | s3cmd put <file> s3://<bucket-name>
|
| Get file from bucket | s3cmd get s3://<bucket-name>/<file-name>
|
| Delete file from bucket | s3cmd [del|rm] s3://<bucket-name>/<file-name>
|
| Show disk usage of buckets | s3cmd du
|
| Show all command available | s3cmd --help
|
Lifecycle
This section holds all sub-sections explaining the lifecycle.
Define the following variables, as they will be used across different lifecycle operations.
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
Lifecycle - Versioning
This section explains how to enable versioning for a s3 bucket.
Lifecycle - Versioning - Get status
To get the current versioning status for a certain bucket, use the following command:
| Text | More text |
|---|---|
| This content | is hidden |
| at first | load time |
# Using awscli
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
# Using s3cmd
s3cmd info s3://${bucket_name}
# Using awscli
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
# Using s3cmd
s3cmd info s3://${bucket_name}
If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.
s3cmd on the other hand will return the following information:
s3://newfreshs3/ (bucket):
[...]
Versioning:none
[...]
If you have versioning disabled, then it will look like this:
# Using awscli
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
# Using s3cmd
s3://newfreshs3/ (bucket):
[...]
Versioning:Suspended
[...]
If you have versioning configured, the response should look something like this:
# Using awscli
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
# Using s3cmd
s3://newfreshs3/ (bucket):
[...]
Versioning:Enabled
[...]
Lifecycle - Versioning - Enable Versioning
To enable versioning use the following command:
# Using awscli
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
# Using s3cmd
s3cmd setversioning s3://${bucket_name} enable
If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).
s3cmd should return the following response:
s3://newfreshs3/: Versioning status updated
Lifecycle - Versioning - Disable Versioning
To disable versioning use the following command:
# Using awscli
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
# Using s3cmd
s3cmd setversioning s3://newfreshs3 disable
If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).
s3cmd should return the following response:
s3://newfreshs3/: Versioning status updated
Lifecycle - Retention policy
This section explains how to add a retention policy to a s3 bucket.
Lifecycle - Retention policy - Get status
To get the bucket policy, run the following command:
# Using awscli
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
# Using s3cmd
s3cmd info s3://${bucket_name}
If the bucket has no retention policy (this is the case for freshly created buckets), then the output of the command will look like this:
# Using awscli
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
# Using s3cmd
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
If the bucket has retention policies, then the output will look similar to this:
# Using the awscli
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
# Using s3cmd
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}
CORS: [...]
ACL: [...]
Lifecycle - Retention policy - Set policy
A typical json formatted policy-file will look something like this:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}
Set a policy for a certain bucket:
#Using the awscli
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
#Using s3cmd
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
If this command ran through successfully using the aws-cli, you will get no response (apart from the exit code 0).
If you ran the s3cmd command, you should get the following response:
s3://yours3bucket/: Policy updated
Lifecycle - Retention policy - Remove Policy
Remove a policy that is no longer wanted:
# Using awscli
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
# Using s3cmd
s3cmd delpolicy s3://${bucket_name}
If this command ran through successfully using the aws-cli, you will get no response (apart from the exit code 0).
If you ran the s3cmd command, you should get the following response:
s3://newfreshs3/: Policy deleted