From 29e60288ded2890e9ddd02ab5a6ae7cff37002b2 Mon Sep 17 00:00:00 2001 From: Ernest Zaslavsky Date: Thu, 19 Dec 2024 17:27:40 +0200 Subject: [PATCH] docs: update the `object_storage.md` and `admin.rst` Added additional options and best practices for AWS authentication. --- docs/dev/object_storage.md | 51 ++++++++++++++++++++++++ docs/operating-scylla/admin.rst | 70 ++++++++++++++++++++++++++++++++- 2 files changed, 120 insertions(+), 1 deletion(-) diff --git a/docs/dev/object_storage.md b/docs/dev/object_storage.md index 9d26ed673c..6053cfd656 100644 --- a/docs/dev/object_storage.md +++ b/docs/dev/object_storage.md @@ -15,6 +15,56 @@ experimental_features: It can also be enabled with `--experimental-features=keyspace-storage-options` command line option when launchgin scylla. +## Configuring AWS S3 access + +You can define endpoint details and authentication tokens in the `object_storage.yaml` file. For example: +```yaml +endpoints: + - name: s3.us-east-1.amazonaws.com + port: 443 + https: true + aws_region: us-east-1 +``` + +### Local/Development Environment + +In a local or development environment, you usually need to add authentication tokens to this file to ensure the client works properly. For instance: +```yaml +endpoints: + - name: s3.us-east-2.amazonaws.com + port: 443 + https: true + aws_region: us-east-2 + aws_access_key_id: EXAMPLE_ACCESS_KEY_ID + aws_secret_access_key: EXAMPLE_SECRET_ACCESS_KEY +``` + +Additionally, you may include an `aws_session_token`, although this is not typically necessary for local or development environments: + +```yaml +endpoints: + - name: s3.us-east-2.amazonaws.com + port: 443 + https: true + aws_region: us-east-2 + aws_access_key_id: EXAMPLE_ACCESS_KEY_ID + aws_secret_access_key: EXAMPLE_SECRET_ACCESS_KEY + aws_session_token: EXAMPLE_TEMPORARY_SESSION_TOKEN +``` +### Important Note + +The examples above are intended for development or local environments. You should *never* use this approach in production. The Scylla S3 client will first attempt to access credentials from the file or environment variables. If it fails to obtain credentials, it will then try to retrieve them from the AWS Security Token Service (STS) or the EC2 Instance Metadata Service. + +For the EC2 Instance Metadata Service to function correctly, no additional configuration is required. However, STS requires the IAM Role ARN to be defined in the `object_storage.yaml` file, as shown below: +```yaml +endpoints: + - name: s3.us-east-1.amazonaws.com + port: 443 + https: true + aws_region: us-east-1 + iam_role_arn: arn:aws:iam::123456789012:instance-profile/my-instance-instance-profile +``` + ## Creating keyspace Sstables location is keyspace-scoped. In order to create a keyspace with S3 @@ -36,6 +86,7 @@ endpoints: aws_region: us-east-2 aws_access_key_id: EXAMPLE_ACCESS_KEY_ID aws_secret_access_key: EXAMPLE_SECRET_ACCESS_KEY + aws_session_token: EXAMPLE_TEMPORARY_SESSION_TOKEN ``` and when creating the keyspace: diff --git a/docs/operating-scylla/admin.rst b/docs/operating-scylla/admin.rst index ab1b473290..65325275df 100644 --- a/docs/operating-scylla/admin.rst +++ b/docs/operating-scylla/admin.rst @@ -104,7 +104,7 @@ Scylla has the ability to communicate directly with S3-compatible storage. This feature enables various functionalities, but requires proper configuration of storage endpoints. -To enable S3-compatible storage features, you need to describe the endpoints +To enable S3-compatible storage features, you need to describe the endpoints and credentials where SSTable files can be stored. This is done using a YAML configuration file. The ``object_storage.yaml`` file should follow this format: @@ -146,6 +146,74 @@ in the same directory as ``scylla.yaml``. You can override this location using t object-storage-config-file: object-storage-config-file.yaml +Configuring AWS S3 access +------------------------- + +You can define endpoint details and authentication tokens in the +``object_storage.yaml`` file. For example: + +.. code:: yaml + + endpoints: + - name: s3.us-east-1.amazonaws.com + port: 443 + https: true + aws_region: us-east-1 + +Local/Development Environment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In a local or development environment, you usually need to add +authentication tokens to this file to ensure the client works properly. +For instance: + +.. code:: yaml + + endpoints: + - name: s3.us-east-2.amazonaws.com + port: 443 + https: true + aws_region: us-east-2 + aws_access_key_id: EXAMPLE_ACCESS_KEY_ID + aws_secret_access_key: EXAMPLE_SECRET_ACCESS_KEY + +Additionally, you may include an ``aws_session_token``, although this is +not typically necessary for local or development environments: + +.. code:: yaml + + endpoints: + - name: s3.us-east-2.amazonaws.com + port: 443 + https: true + aws_region: us-east-2 + aws_access_key_id: EXAMPLE_ACCESS_KEY_ID + aws_secret_access_key: EXAMPLE_SECRET_ACCESS_KEY + aws_session_token: EXAMPLE_TEMPORARY_SESSION_TOKEN + +Important Note +^^^^^^^^^^^^^^^ + +The examples above are intended for development or local environments. +You should *never* use this approach in production. The Scylla S3 client +will first attempt to access credentials from the file or environment +variables. If it fails to obtain credentials, it will then try to +retrieve them from the AWS Security Token Service (STS) or the EC2 +Instance Metadata Service. + +For the EC2 Instance Metadata Service to function correctly, no +additional configuration is required. However, STS requires the IAM Role +ARN to be defined in the ``object_storage.yaml`` file, as shown below: + +.. code:: yaml + + endpoints: + - name: s3.us-east-1.amazonaws.com + port: 443 + https: true + aws_region: us-east-1 + iam_role_arn: arn:aws:iam::123456789012:instance-profile/my-instance-instance-profile + .. _admin-compression: Compression