MediaWiki - User contributions [en]

S3 buckets

2025-03-06T09:01:04Z

Sst-la1: /* Credential pair */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
{{Note|title=Information|content=Unfortunately, ec2 credentials can not be managed via the graphical OpenStack dashboard as of this moment.}}

In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==
Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To create a new s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To create a new s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To create a new s3-bucket using the OpenStack dashboard, follow this section.

First go to <code>Object Store -> Containers</code> and press <code>+ Container</code>: 

[[File:S3-buckets_-_Lifecycle_Create_bucket_01.png|1000px]]

 
After doing that, enter a name that isn't taken yet, press <code>Public</code> or <code>Not public</code> and submit. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_02.png|1000px]]

 
The s3-bucket has been created successfully if it shows up in the list of s3-buckets. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_03.png|1000px]]

 
There you also have the possibility to create new folders and upload files within that s3-bucket.

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2025-03-06T09:00:49Z

Sst-la1: /* Credential pair - Create */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==
Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To create a new s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To create a new s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To create a new s3-bucket using the OpenStack dashboard, follow this section.

First go to <code>Object Store -> Containers</code> and press <code>+ Container</code>: 

[[File:S3-buckets_-_Lifecycle_Create_bucket_01.png|1000px]]

 
After doing that, enter a name that isn't taken yet, press <code>Public</code> or <code>Not public</code> and submit. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_02.png|1000px]]

 
The s3-bucket has been created successfully if it shows up in the list of s3-buckets. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_03.png|1000px]]

 
There you also have the possibility to create new folders and upload files within that s3-bucket.

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2025-03-06T08:59:59Z

Sst-la1: /* Credential pair - Create */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==
{{Note|title=Information|content=Unfortunately, ec2 credentials can not be created via the graphical OpenStack dashboard as of this moment.}}

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To create a new s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To create a new s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To create a new s3-bucket using the OpenStack dashboard, follow this section.

First go to <code>Object Store -> Containers</code> and press <code>+ Container</code>: 

[[File:S3-buckets_-_Lifecycle_Create_bucket_01.png|1000px]]

 
After doing that, enter a name that isn't taken yet, press <code>Public</code> or <code>Not public</code> and submit. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_02.png|1000px]]

 
The s3-bucket has been created successfully if it shows up in the list of s3-buckets. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_03.png|1000px]]

 
There you also have the possibility to create new folders and upload files within that s3-bucket.

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2025-03-06T08:59:42Z

Sst-la1: /* Credential pair - Create */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==
{{Note|title=Information|content=Unfortunately, ec2 credentials can not be created via the graphical OpenStack Dashboard as of this moment.}}

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To create a new s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To create a new s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To create a new s3-bucket using the OpenStack dashboard, follow this section.

First go to <code>Object Store -> Containers</code> and press <code>+ Container</code>: 

[[File:S3-buckets_-_Lifecycle_Create_bucket_01.png|1000px]]

 
After doing that, enter a name that isn't taken yet, press <code>Public</code> or <code>Not public</code> and submit. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_02.png|1000px]]

 
The s3-bucket has been created successfully if it shows up in the list of s3-buckets. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_03.png|1000px]]

 
There you also have the possibility to create new folders and upload files within that s3-bucket.

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2025-03-06T08:59:33Z

Sst-la1: /* Credential pair - Create */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==
{{Note|title=Information:|content=Unfortunately, ec2 credentials can not be created via the graphical OpenStack Dashboard as of this moment.}}

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To create a new s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To create a new s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To create a new s3-bucket using the OpenStack dashboard, follow this section.

First go to <code>Object Store -> Containers</code> and press <code>+ Container</code>: 

[[File:S3-buckets_-_Lifecycle_Create_bucket_01.png|1000px]]

 
After doing that, enter a name that isn't taken yet, press <code>Public</code> or <code>Not public</code> and submit. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_02.png|1000px]]

 
The s3-bucket has been created successfully if it shows up in the list of s3-buckets. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_03.png|1000px]]

 
There you also have the possibility to create new folders and upload files within that s3-bucket.

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2025-03-06T08:59:10Z

Sst-la1: /* Credential pair - Create */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==
{{Note|title=Information|content=Unfortunately, ec2 credentials can not be created via the graphical OpenStack Dashboard as of this moment.}}

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To create a new s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To create a new s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To create a new s3-bucket using the OpenStack dashboard, follow this section.

First go to <code>Object Store -> Containers</code> and press <code>+ Container</code>: 

[[File:S3-buckets_-_Lifecycle_Create_bucket_01.png|1000px]]

 
After doing that, enter a name that isn't taken yet, press <code>Public</code> or <code>Not public</code> and submit. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_02.png|1000px]]

 
The s3-bucket has been created successfully if it shows up in the list of s3-buckets. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_03.png|1000px]]

 
There you also have the possibility to create new folders and upload files within that s3-bucket.

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2025-03-06T08:58:39Z

Sst-la1: /* Credential pair - Create */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==
{{Note|title=Info|content=Unfortunately, ec2 credentials can not be created via the graphical OpenStack Dashboard as of this moment.}}

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To create a new s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To create a new s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To create a new s3-bucket using the OpenStack dashboard, follow this section.

First go to <code>Object Store -> Containers</code> and press <code>+ Container</code>: 

[[File:S3-buckets_-_Lifecycle_Create_bucket_01.png|1000px]]

 
After doing that, enter a name that isn't taken yet, press <code>Public</code> or <code>Not public</code> and submit. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_02.png|1000px]]

 
The s3-bucket has been created successfully if it shows up in the list of s3-buckets. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_03.png|1000px]]

 
There you also have the possibility to create new folders and upload files within that s3-bucket.

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2025-03-06T08:55:51Z

Sst-la1: /* Credential pair - Create */

S3 buckets

2025-03-06T08:37:06Z

Sst-la1: /* Credential pair - Create */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

<tabber>
|-| Openstack-CLI =
Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>
|-| OpenStack Dashboard =

</tabber>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To create a new s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To create a new s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To create a new s3-bucket using the OpenStack dashboard, follow this section.

First go to <code>Object Store -> Containers</code> and press <code>+ Container</code>: 

[[File:S3-buckets_-_Lifecycle_Create_bucket_01.png|1000px]]

 
After doing that, enter a name that isn't taken yet, press <code>Public</code> or <code>Not public</code> and submit. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_02.png|1000px]]

 
The s3-bucket has been created successfully if it shows up in the list of s3-buckets. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_03.png|1000px]]

 
There you also have the possibility to create new folders and upload files within that s3-bucket.

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-12-24T10:36:14Z

Sst-la1: /* Lifecycle - Create bucket */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To create a new s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To create a new s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To create a new s3-bucket using the OpenStack dashboard, follow this section.

First go to <code>Object Store -> Containers</code> and press <code>+ Container</code>: 

[[File:S3-buckets_-_Lifecycle_Create_bucket_01.png|1000px]]

 
After doing that, enter a name that isn't taken yet, press <code>Public</code> or <code>Not public</code> and submit. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_02.png|1000px]]

 
The s3-bucket has been created successfully if it shows up in the list of s3-buckets. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_03.png|1000px]]

 
There you also have the possibility to create new folders and upload files within that s3-bucket.

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-12-24T10:35:47Z

Sst-la1: /* Lifecycle - Create bucket */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To create a new s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To create a new s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To create a new s3-bucket using the OpenStack dashboard, follow this section.

First go to <code>Object Store -> Containers</code> and press <code>+ Container</code>: 

[[File:S3-buckets_-_Lifecycle_Create_bucket_01.png|1000px]]

 
After doing that, enter a name that isn't taken yet, press <code>Public</code> or <code>Not public</code> and submit. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_02.png|1000px]]

 
The s3-bucket has been created successfully if it shows up in the list of s3-buckets. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_02.png|1000px]]

 
There you also have the possibility to create new folders and upload files within that s3-bucket.

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-12-24T10:32:36Z

Sst-la1: /* Lifecycle - Create bucket */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To create a new s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To create a new s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To create a new s3-bucket using the OpenStack dashboard, follow this section.

First go to <code>Object Store -> Containers</code> and press <code>+ Container</code>: 

[[File:S3-buckets_-_Lifecycle_Create_bucket_01.png|1000px]]

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-12-24T10:31:59Z

Sst-la1: /* Lifecycle - Create bucket */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To create a new s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To create a new s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To create a new s3-bucket using the OpenStack dashboard, follow this section.

First go to <code>Object Store -> Containers</code>: 

[[File:S3-buckets_-_Lifecycle_Create_bucket_01.png|1000px]]

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-12-24T10:31:02Z

Sst-la1: /* Lifecycle - Create bucket */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To create a new s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To create a new s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To create a new s3-bucket using the OpenStack dashboard, follow this section. 

[[File:S3-buckets_-_Lifecycle_Create_bucket_01.png|1000px]]

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-12-24T10:30:43Z

Sst-la1: /* Lifecycle - Create bucket */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To create a new s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To create a new s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To create a new s3-bucket using the OpenStack dashboard, follow this section.

[[File:S3-buckets_-_Lifecycle_Create_bucket_01.png|1000px]]

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-12-24T10:30:20Z

Sst-la1: /* Lifecycle - Create bucket */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To create a new s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To create a new s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To create a new s3-bucket using the OpenStack dashboard, follow this section.

[[File:S3-buckets_-_Lifecycle_Create_bucket_01.png|1000px]]

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-12-24T10:23:26Z

Sst-la1: /* Lifecycle - Create bucket */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To create a new s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To create a new s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To create a new s3-bucket using the OpenStack dashboard, follow this section.

[[File:S3-buckets_-_Lifecycle_Create_bucket_01.png|1000px]]

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-12-24T10:23:10Z

Sst-la1: /* Lifecycle - Create bucket */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To create a new s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To create a new s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To create a new s3-bucket using the OpenStack dashboard, follow this section.
[[File:S3-buckets_-_Lifecycle_Create_bucket_01.png|1000px]]

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-12-24T10:22:57Z

Sst-la1: /* Lifecycle - Create bucket */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To create a new s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To create a new s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To create a new s3-bucket using the OpenStack dashboard, follow this section.
[[File:S3-buckets_-_Lifecycle_Create_bucket_01.png|thumb]]

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-12-24T10:22:27Z

Sst-la1: /* Lifecycle - Create bucket */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To create a new s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To create a new s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To create a new s3-bucket using the OpenStack dashboard, follow this section.
[[File:S3-buckets - Lifecycle Create bucket 01|thumb]]

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-12-24T10:21:06Z

Sst-la1: /* Lifecycle - Create bucket */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To create a new s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To create a new s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-12-24T10:19:52Z

Sst-la1: /* Lifecycle - Create bucket */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

|-| OpenStack Dashboard =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

File:S3-buckets - Lifecycle Create bucket 03.png

2024-12-24T10:19:14Z

Sst-la1:

File:S3-buckets - Lifecycle Create bucket 02.png

2024-12-24T10:18:58Z

Sst-la1:

File:S3-buckets - Lifecycle Create bucket 01.png

2024-12-24T10:18:47Z

Sst-la1:

S3 buckets

2024-11-26T14:55:22Z

Sst-la1: /* Lifecycle - Versioning - Disable Versioning */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:55:05Z

Sst-la1: /* Lifecycle - Versioning - Enable Versioning */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get status]] command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:54:10Z

Sst-la1: /* Lifecycle - Versioning - Enable Versioning */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the [[#Lifecycle_-_Versioning_-_Get_status|Get status]] command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:53:34Z

Sst-la1: /* Lifecycle - Versioning - Enable Versioning */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

If you then run the ''-command you will get an output like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:50:23Z

Sst-la1: /* Lifecycle - Policy - Get status */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash" highlight=8>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:48:47Z

Sst-la1: /* Lifecycle - Versioning - Get status */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang=text highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Suspended
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:47:15Z

Sst-la1: /* Lifecycle - Versioning - Get status */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:none
[...]
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Suspended
[...]
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Enabled
[...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:46:29Z

Sst-la1: /* Lifecycle - Versioning - Get status */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="text" highlight=5>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:none
[...]
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Suspended
[...]
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Enabled
[...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:46:11Z

Sst-la1: /* Lifecycle - Versioning - Get status */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json" highlight=4>
s3://test77/ (bucket):
Location: location
Payer: BucketOwner
Ownership: none
Versioning:Enabled
Expiration rule: none
Block Public Access: none
Policy: none
CORS: none
ACL: x: FULL_CONTROL
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:none
[...]
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Suspended
[...]
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Enabled
[...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:42:41Z

Sst-la1: /* Lifecycle - Object lock - Set retention policy */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:none
[...]
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Suspended
[...]
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Enabled
[...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:42:19Z

Sst-la1: /* Lifecycle - Object lock - Get retention policy */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:none
[...]
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Suspended
[...]
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Enabled
[...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

If you already have a retention policy set, then the output might look something like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 3
}
}
}
}
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:40:10Z

Sst-la1: /* Lifecycle - Object lock - Set retention policy */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:none
[...]
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Suspended
[...]
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Enabled
[...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, use the [[#Lifecycle_-_Object_lock_-_Get_retention_policy|get-object-lock-configuration command]].
The output should then look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:38:36Z

Sst-la1: /* Lifecycle - Object lock - Set retention policy */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:none
[...]
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Suspended
[...]
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Enabled
[...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To set the retention time of a s3-bucket to a certain amount of days, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-object-lock-configuration --bucket ${bucket_name} --object-lock-configuration 'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=3}}'
</syntaxhighlight>

'''Output:'''

The command will give you no output (apart from the status code 0).

To check the retention policy of the s3-bucket, go to the following section
The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:34:11Z

Sst-la1: /* Lifecycle - Object lock - Set retention policy */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:none
[...]
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Suspended
[...]
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Enabled
[...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===
Use the following commands to set the retention policy of a s3-bucket.

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:33:16Z

Sst-la1: /* Lifecycle - Object lock - Get retention policy */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:none
[...]
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Suspended
[...]
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Enabled
[...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="text">
An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Unknown
</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:32:26Z

Sst-la1: /* Lifecycle - Object lock */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:none
[...]
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Suspended
[...]
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Enabled
[...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the '--object-lock-enabled-for-bucket'-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="bash">

</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:31:58Z

Sst-la1: /* Lifecycle - Object lock - Get retention policy */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:none
[...]
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Suspended
[...]
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Enabled
[...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the ''-flag ([[#Lifecycle_-_Create_bucket_with_object_locking|Create bucket with object lock enabled]]).
<syntaxhighlight lang="bash">

</syntaxhighlight>

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:30:45Z

Sst-la1: /* Lifecycle - Object lock */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:none
[...]
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Suspended
[...]
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Enabled
[...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==
This section covers how to manage object locks and set retention policies.

=== Lifecycle - Object lock - Get retention policy ===
Use the following commands to retrieve the current retention policy of a s3-bucket.

<tabber>
|-| Aws-cli =
To get the retention policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

The output should look like this:
<syntaxhighlight lang="json">
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
</syntaxhighlight>

If it looks like this instead, then you created the s3-bucket without the ''-flag (Create bucket with object lock enabled).

|-| S3cmd =
Not supported by s3cmd.
</tabber>

=== Lifecycle - Object lock - Set retention policy ===

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:26:56Z

Sst-la1: /* Lifecycle - Retention policy */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:none
[...]
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Suspended
[...]
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Enabled
[...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Object lock ==

=== Lifecycle - Object lock - Get retention policy ===
=== Lifecycle - Object lock - Set retention policy ===

aws --endpoint-url ${endpoint_url} s3api get-object-lock-configuration --bucket ${bucket_name}

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:24:24Z

Sst-la1: /* Lifecycle - Policy */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:none
[...]
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Suspended
[...]
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Enabled
[...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Retention policy ==

== Lifecycle - Policy ==
This section explains how to add a policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
</syntaxhighlight>

If the bucket has a policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:23:48Z

Sst-la1: /* Lifecycle - Policy */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:none
[...]
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Suspended
[...]
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Enabled
[...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Retention policy ==

== Lifecycle - Policy ==
This section explains how to add a retention policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no retention policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a retention policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no retention policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
</syntaxhighlight>

If the bucket has a retention policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T14:23:18Z

Sst-la1: /* Lifecycle - Retention policy */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:none
[...]
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Suspended
[...]
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Enabled
[...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Policy ==
This section explains how to add a retention policy to a s3 bucket.

=== Lifecycle - Policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no retention policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a retention policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no retention policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
</syntaxhighlight>

If the bucket has a retention policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T13:52:28Z

Sst-la1: /* Lifecycle - Create bucket with object locking */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
s3cmd doesn't support object-locks.
</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:none
[...]
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Suspended
[...]
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Enabled
[...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Retention policy ==
This section explains how to add a retention policy to a s3 bucket.

=== Lifecycle - Retention policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no retention policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a retention policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no retention policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
</syntaxhighlight>

If the bucket has a retention policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Retention policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Retention policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T13:51:36Z

Sst-la1: /* Lifecycle - Create bucket with object locking */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:none
[...]
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Suspended
[...]
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Enabled
[...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Retention policy ==
This section explains how to add a retention policy to a s3 bucket.

=== Lifecycle - Retention policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no retention policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a retention policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no retention policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
</syntaxhighlight>

If the bucket has a retention policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Retention policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Retention policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]

S3 buckets

2024-11-26T13:48:35Z

Sst-la1: /* Lifecycle - Create bucket */

= Overview =
This page describes the creation and management of S3 buckets in our OpenStack-based stoney cloud.

= Credential pair =
In order to use the S3 API you have to create EC2 (Amazon Elastic Compute Cloud) credentials using the OpenStack Keystone service.

This section will guide you through the creation process in our OpenStack-based cloud.

== Credential pair - Create ==

Create new EC2 credentials in OpenStack using the OpenStack-CLI:
<syntaxhighlight lang="bash">
openstack ec2 credentials create
</syntaxhighlight>

This will give you an output in the following format:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Show ==

If you ever need to look the credentials up again, use the following command:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials show ${access_id}
</syntaxhighlight>

This will give you an output formatted like this:
<syntaxhighlight lang="text">
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access | tpvx3i0gk5rf4duomnr7davjxl517z9c |
| links | {'self': 'https://api.os.stoney-cloud.com:5000/v3/users/tpvx3i0gk5rf4duomnr7davjxl517z9c/credentials/OS-EC2/tpvx3i0gk5rf4duomnr7davjxl517z9c'} |
| project_id | hw3rr6x6ktyuv7erwpuyxbijihx1phdw |
| secret | 6lifckxv1005z60csekl7qynwxwbv3re |
| trust_id | None |
| user_id | tpvx3i0gk5rf4duomnr7davjxl517z9c |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
</syntaxhighlight>

== Credential pair - Delete ==
If you need to delete your credentials, you can so like this:
<syntaxhighlight lang="bash">
access_id=tpvx3i0gk5rf4duomnr7davjxl517z9c
openstack ec2 credentials delete ${access_id}
</syntaxhighlight>

When running 'delete' you should get no response apart from the status code 0.

= General usage =
When using the S3 technology, you have different possible cli-tools.
The most popular implementations are:
* aws
* s3cmd

This page focuses on the usage of those two implementations.

== General usage - Connect ==
=== General usage - Connect - AWS client ===
This section explains the general usage such as configuring the connection using the AWS-client.
==== General usage - Connect - AWS client - Installation ====
Install the awscli using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install awscli
# Ubuntu/Debian
sudo apt install awscli
# Alpine Linux
sudo apk add aws-cli
# Arch Linux
sudo pacman -S aws-cli
</syntaxhighlight>

==== General usage - Connect - AWS client - Configuration ====
After installing the awscli package, you can configure it like so:
<syntaxhighlight lang="bash">
aws configure
</syntaxhighlight>

The configuration helper will prompt you to enter the following information:
<syntaxhighlight lang="bash">
AWS Access Key ID [None]: tpvx3i0gk5rf4duomnr7davjxl517z9c # access (from EC2 credentials)
AWS Secret Access Key [None]: 6lifckxv1005z60csekl7qynwxwbv3re # secret (from EC2 credentials)
Default region name [None]: # leave empty
Default output format [None]: json # set to json
</syntaxhighlight>

This will then create config files on your machine in the following locations:
* ~/.aws/config
* ~/.aws/credentials

==== General usage - Connect - AWS client - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">aws s3api list-buckets</syntaxhighlight> or <syntaxhighlight lang="bash">aws s3 ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">aws s3api create-bucket <bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">aws s3api delete-bucket --bucket <bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">aws s3api list-objects --bucket <bucket-name></syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">aws s3api help</syntaxhighlight>
|}

=== General usage - Connect - S3cmd ===
This section explains the general usage such as configuring the connection using the S3cmd-client.

==== General usage - Connect - S3cmd - Installation ====
Install the s3cmd using your favorite package manager:
<syntaxhighlight lang="bash">
# Fedora/RHEL
sudo dnf install s3cmd
# Ubuntu/Debian
sudo apt install s3cmd
# Alpine Linux
sudo apk add s3cmd
# Arch Linux
sudo pacman -S s3cmd
</syntaxhighlight>

==== General usage - Connect - S3cmd - Configuration ====
To configure s3cmd, create a configuration file like so:
<syntaxhighlight lang="bash">
# Create file
touch ~/.s3cfg

# Edit file
vim ~/.s3cfg
</syntaxhighlight>

The configuration file should include the following options:
<syntaxhighlight lang="bash">
access_key = <access> # replace with your access key of the ec2 credential
secret_key = <secret> # replace with your secret key of the ec2 credential
host_base = api.os.stoney-cloud.com:9000
host_bucket = api.os.stoney-cloud.com:9000
</syntaxhighlight>

==== General usage - Connect - S3cmd - Cheatsheet ====
Short overview of available commands when using s3cmd:
{| class="wikitable"
|+ Cheatsheet
|-
! Description !! Command
|-
| Show available buckets || <syntaxhighlight lang="bash">s3cmd ls</syntaxhighlight>
|-
| Create a bucket || <syntaxhighlight lang="bash">s3cmd mb s3://<bucket-name></syntaxhighlight>
|-
| Delete a bucket || <syntaxhighlight lang="bash">s3cmd rb s3://<bucket-name></syntaxhighlight>
|-
| Show content of a bucket || <syntaxhighlight lang="bash">s3cmd ls s3://<bucket-name></syntaxhighlight>
|-
| Put file into bucket || <syntaxhighlight lang="bash">s3cmd put <file> s3://<bucket-name></syntaxhighlight>
|-
| Get file from bucket || <syntaxhighlight lang="bash">s3cmd get s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Delete file from bucket || <syntaxhighlight lang="bash">s3cmd [del|rm] s3://<bucket-name>/<file-name></syntaxhighlight>
|-
| Show disk usage of buckets || <syntaxhighlight lang="bash">s3cmd du</syntaxhighlight>
|-
| Show all command available || <syntaxhighlight lang="bash">s3cmd --help</syntaxhighlight>
|}

= Lifecycle =
This section holds all sub-sections explaining the lifecycle.

Define the following variables, as they will be used across different lifecycle operations.
<syntaxhighlight lang="bash">
endpoint_url=https://api.os.stoney-cloud.com:9000
bucket_name=<bucket-name>
</syntaxhighlight>

== Lifecycle - Create bucket ==
This section explains how to create a new s3 bucket.

=== Lifecycle - Create bucket ===
The following commands explain how to create a normal s3-bucket.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the follwing command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3 ls | grep ${bucket_name}
</syntaxhighlight>

The output should then look something like this:
<syntaxhighlight lang="text">
2024-11-26 14:46:28 test77
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd mb s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
Bucket 's3://test76/' created
</syntaxhighlight>

</tabber>

=== Lifecycle - Create bucket with object locking ===
To create a s3-bucket with support for object-locking, use the following commands.
<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api create-bucket --bucket ${bucket_name} --object-lock-enabled-for-bucket
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Versioning ==
This section explains how to enable versioning for a s3 bucket.

=== Lifecycle - Versioning - Get status ===
To get the current versioning status for a certain bucket, use the following commands.

<tabber>
|-| Aws-cli =
To retrieve the status of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-versioning --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, the aws-cli command should return nothing.

If you have versioning disabled, it will look like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To retrieve the status of a s3-bucket using the s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd info s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you haven't configured versioning for that particular bucket yet, s3cmd will return the following information:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:none
[...]
</syntaxhighlight>

If you have versioning disabled, it will look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Suspended
[...]
</syntaxhighlight>

If you have versioning configured, it should look similar to this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Versioning:Enabled
[...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Versioning - Enable Versioning ===
To enable versioning use the following commands.

<tabber>
|-| Aws-cli =
To enable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Versioning - Disable Versioning ===
To disable versioning use the following commands.

<tabber>
|-| Aws-cli =
To disable versioning of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Suspended
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

You can check if the command was successful, by running the [[#Lifecycle - Versioning - Get status|Get-status]]-command. The output should then look something like this:
<syntaxhighlight lang="json">
{
"Status": "Suspended",
"MFADelete": "Disabled"
}
</syntaxhighlight>

|-| S3cmd =
To enable versioning of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://newfreshs3 disable
</syntaxhighlight>

'''Output:'''

s3cmd should return the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Versioning status updated
</syntaxhighlight>

</tabber>

== Lifecycle - Retention policy ==
This section explains how to add a retention policy to a s3 bucket.

=== Lifecycle - Retention policy - Get status ===
To get the bucket policy, run the following commands.

<tabber>
|-| Aws-cli =
To get the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api get-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the bucket has no retention policy, then the output should look something like this:
<syntaxhighlight lang="text">
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</syntaxhighlight>

if the bucket has a retention policy, then the output should look something like this:
<syntaxhighlight lang="json">
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"AWS\": [\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:root\",\n \"arn:aws:iam::tpvx3i0gk5rf4duomnr7davjxl517z9c:user/testuser\"\n ]},\n \"Action\": [\n \"s3:ListBucket\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:GetObject\"\n ],\n \"Resource\": [\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket/*\",\n \"arn:aws:s3:::tpvx3i0gk5rf4duomnr7davjxl517z9c/backup-Y-bucket\"\n ]\n }]\n}\n\n"
}
</syntaxhighlight>

|-| S3cmd =
To get the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setversioning s3://${bucket_name} enable
</syntaxhighlight>

'''Output:'''

If the bucket has no retention policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://newfreshs3/ (bucket):
[...]
Policy: none
[...]
</syntaxhighlight>

If the bucket has a retention policy, then the output will look something like this:
<syntaxhighlight lang="bash">
s3://secondtest/ (bucket):
[...]
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

CORS: [...]
ACL: [...]
</syntaxhighlight>
</tabber>

=== Lifecycle - Retention policy - Set policy ===
A typical json formatted policy-file will look something like this:
<syntaxhighlight lang="json">
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": [
"arn:aws:iam::x:root",
"arn:aws:iam::x:user/testuser"
]},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::x/backup-Y-bucket/*",
"arn:aws:s3:::x/backup-Y-bucket"
]
}]
}

</syntaxhighlight>

The following commands are used to set the policy for a certain bucket.
<tabber>
|-| Aws-cli =
To set the bucket policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api put-bucket-policy --bucket ${bucket_name} --policy file://<policy_file>.json
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To set the bucket policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd setpolicy <policy_file>.json s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://yours3bucket/: Policy updated
</syntaxhighlight>

</tabber>

=== Lifecycle - Retention policy - Remove Policy ===
The following commands are used to remove a policy that is no longer wanted:

<tabber>
|-| Aws-cli =
To remove the policy of a s3-bucket using the aws-cli, use the following command:
<syntaxhighlight lang="bash">
aws --endpoint-url ${endpoint_url} s3api delete-bucket-policy --bucket ${bucket_name}
</syntaxhighlight>

'''Output:'''

If the aws-cli command ran through successfully, you will get no response (apart from the exit code 0).

|-| S3cmd =
To remove the policy of a s3-bucket using s3cmd, use the following command:
<syntaxhighlight lang="bash">
s3cmd delpolicy s3://${bucket_name}
</syntaxhighlight>

'''Output:'''

If you ran the s3cmd command, you should get the following response:
<syntaxhighlight lang="bash">
s3://newfreshs3/: Policy deleted
</syntaxhighlight>

</tabber>

[[Category: CLI]]
[[Category: Object_store]]