docs: update the object_storage.md and admin.rst

Added additional options and best practices for AWS authentication.
This commit is contained in:
Ernest Zaslavsky
2024-12-19 17:27:40 +02:00
parent dee4fc7150
commit 29e60288de
2 changed files with 120 additions and 1 deletions

View File

@@ -15,6 +15,56 @@ experimental_features:
It can also be enabled with `--experimental-features=keyspace-storage-options`
command line option when launchgin scylla.
## Configuring AWS S3 access
You can define endpoint details and authentication tokens in the `object_storage.yaml` file. For example:
```yaml
endpoints:
- name: s3.us-east-1.amazonaws.com
port: 443
https: true
aws_region: us-east-1
```
### Local/Development Environment
In a local or development environment, you usually need to add authentication tokens to this file to ensure the client works properly. For instance:
```yaml
endpoints:
- name: s3.us-east-2.amazonaws.com
port: 443
https: true
aws_region: us-east-2
aws_access_key_id: EXAMPLE_ACCESS_KEY_ID
aws_secret_access_key: EXAMPLE_SECRET_ACCESS_KEY
```
Additionally, you may include an `aws_session_token`, although this is not typically necessary for local or development environments:
```yaml
endpoints:
- name: s3.us-east-2.amazonaws.com
port: 443
https: true
aws_region: us-east-2
aws_access_key_id: EXAMPLE_ACCESS_KEY_ID
aws_secret_access_key: EXAMPLE_SECRET_ACCESS_KEY
aws_session_token: EXAMPLE_TEMPORARY_SESSION_TOKEN
```
### Important Note
The examples above are intended for development or local environments. You should *never* use this approach in production. The Scylla S3 client will first attempt to access credentials from the file or environment variables. If it fails to obtain credentials, it will then try to retrieve them from the AWS Security Token Service (STS) or the EC2 Instance Metadata Service.
For the EC2 Instance Metadata Service to function correctly, no additional configuration is required. However, STS requires the IAM Role ARN to be defined in the `object_storage.yaml` file, as shown below:
```yaml
endpoints:
- name: s3.us-east-1.amazonaws.com
port: 443
https: true
aws_region: us-east-1
iam_role_arn: arn:aws:iam::123456789012:instance-profile/my-instance-instance-profile
```
## Creating keyspace
Sstables location is keyspace-scoped. In order to create a keyspace with S3
@@ -36,6 +86,7 @@ endpoints:
aws_region: us-east-2
aws_access_key_id: EXAMPLE_ACCESS_KEY_ID
aws_secret_access_key: EXAMPLE_SECRET_ACCESS_KEY
aws_session_token: EXAMPLE_TEMPORARY_SESSION_TOKEN
```
and when creating the keyspace:

View File

@@ -104,7 +104,7 @@ Scylla has the ability to communicate directly with S3-compatible storage. This
feature enables various functionalities, but requires proper configuration of
storage endpoints.
To enable S3-compatible storage features, you need to describe the endpoints
To enable S3-compatible storage features, you need to describe the endpoints and credentials
where SSTable files can be stored. This is done using a YAML configuration file.
The ``object_storage.yaml`` file should follow this format:
@@ -146,6 +146,74 @@ in the same directory as ``scylla.yaml``. You can override this location using t
object-storage-config-file: object-storage-config-file.yaml
Configuring AWS S3 access
-------------------------
You can define endpoint details and authentication tokens in the
``object_storage.yaml`` file. For example:
.. code:: yaml
endpoints:
- name: s3.us-east-1.amazonaws.com
port: 443
https: true
aws_region: us-east-1
Local/Development Environment
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In a local or development environment, you usually need to add
authentication tokens to this file to ensure the client works properly.
For instance:
.. code:: yaml
endpoints:
- name: s3.us-east-2.amazonaws.com
port: 443
https: true
aws_region: us-east-2
aws_access_key_id: EXAMPLE_ACCESS_KEY_ID
aws_secret_access_key: EXAMPLE_SECRET_ACCESS_KEY
Additionally, you may include an ``aws_session_token``, although this is
not typically necessary for local or development environments:
.. code:: yaml
endpoints:
- name: s3.us-east-2.amazonaws.com
port: 443
https: true
aws_region: us-east-2
aws_access_key_id: EXAMPLE_ACCESS_KEY_ID
aws_secret_access_key: EXAMPLE_SECRET_ACCESS_KEY
aws_session_token: EXAMPLE_TEMPORARY_SESSION_TOKEN
Important Note
^^^^^^^^^^^^^^^
The examples above are intended for development or local environments.
You should *never* use this approach in production. The Scylla S3 client
will first attempt to access credentials from the file or environment
variables. If it fails to obtain credentials, it will then try to
retrieve them from the AWS Security Token Service (STS) or the EC2
Instance Metadata Service.
For the EC2 Instance Metadata Service to function correctly, no
additional configuration is required. However, STS requires the IAM Role
ARN to be defined in the ``object_storage.yaml`` file, as shown below:
.. code:: yaml
endpoints:
- name: s3.us-east-1.amazonaws.com
port: 443
https: true
aws_region: us-east-1
iam_role_arn: arn:aws:iam::123456789012:instance-profile/my-instance-instance-profile
.. _admin-compression:
Compression