alternator: document the state of tablet support in Alternator

In commit c24bc3b we decided that creating a new table in Alternator
will by default use vnodes - not tablets - because of all the missing
features in our tablets implementation that are important for
Alternator, namely - LWT, CDC and Alternator TTL.

We never documented this, or the fact that we support a tag
`experimental:initial_tablets` which allows to override this decision
and create an Alternator table using tablets. We also never documented
what exactly doesn't work when Alternator uses tablet.

This patch adds the missing documentation in docs/alternator/new-apis.md
(which is a good place for describing the `experimental:initial_tablets`
tag). The patch also adds a new test file, test_tablets.py, which
includes tests for all the statements made in the document regarding
how `experimental:initial_tablets` works and what works or doesn't
work when tablets are enabled.

Two existing tests - for TTL and Streams non-support with tablets -
are moved to the new test file.

When the tablets feature will finally be completed, both the document
and the tests will need to be modified (some of the tests should be
outright deleted). But it seems this will not happen for at least
several months, and that is too long to wait without accurate
documentation.

Fixes #21629

Signed-off-by: Nadav Har'El <nyh@scylladb.com>

Closes scylladb/scylladb#22462

(cherry picked from commit c0821842de)

Closes scylladb/scylladb#23298

docs/alternator/new-apis.md

@@ -144,3 +144,46 @@ If a certain data center or rack has no functional nodes, or doesn't even
 exist, an empty list (`[]`) is returned by the `/localnodes` request.
 A client should be prepared to consider expanding the node search to an
 entire data center, or other data centers, in that case.
+
+## Tablets
+"Tablets" are ScyllaDB's new approach to replicating data across a cluster.
+It replaces the older approach which was named "vnodes". Compared to vnodes,
+tablets are smaller pieces of tables that are easier to move between nodes,
+and allow for faster growing or shrinking of the cluster when needed.
+
+In this version, tablet support is incomplete and not all of the features
+which Alternator needs are supported with tablets. So currently, new
+Alternator tables default to using vnodes - not tablets.
+
+However, if you do want to create an Alternator table which uses tablets,
+you can do this by specifying the `experimental:initial_tablets` tag in
+the CreateTable operation. The value of this tag can be:
+
+* Any valid integer as the value of this tag enables tablets.
+  Typically the number "0" is used - which tells ScyllaDB to pick a reasonable
+  number of initial tablets. But any other number can be used, and this
+  number overrides the default choice of initial number of tablets.
+
+* Any non-integer value - e.g., the string "none" - creates the table
+  without tablets - i.e., using vnodes.
+
+The `experimental:initial_tablets` tag only has any effect while creating
+a new table with CreateTable - changing it later has no effect.
+
+Because the tablets support is incomplete, when tablets are enabled for an
+Alternator table, the following features will not work for this table:
+
+* The table must have one of the write isolation modes which does not
+  not use LWT, because it's not supported with tablets. The allowed write
+  isolation modes are `forbid_rmw` or `unsafe_rmw`.
+  Setting the isolation mode to `always_use_lwt` will succeed, but the writes
+  themselves will fail with an InternalServerError. At that point you can
+  still change the write isolation mode of the table to a supported mode.
+  See <https://github.com/scylladb/scylladb/issues/18068>.
+
+* Enabling TTL with UpdateTableToLive doesn't work (results in an error).
+  See <https://github.com/scylladb/scylla/issues/16567>.
+
+* Enabling Streams with CreateTable or UpdateTable doesn't work
+  (results in an error).
+  See <https://github.com/scylladb/scylla/issues/16317>.

test/alternator/test_streams.py

@@ -23,26 +23,6 @@ from test.alternator.util import unique_table_name, create_test_table, new_test_
 # using the following tags when creating each table below:
 TAGS = [{'Key': 'experimental:initial_tablets', 'Value': 'none'}]
 
-# Before Alternator Streams is supported with tablets (#16317), let's verify
-# that enabling Streams results in an orderly error. This test should be
-# deleted when #16317 is fixed.
-def test_streams_enable_error_with_tablets(dynamodb, scylla_only):
-    # Test attempting to create a table already with streams
-    with pytest.raises(ClientError, match='ValidationException.*tablets'):
-        with new_test_table(dynamodb,
-            Tags=[{'Key': 'experimental:initial_tablets', 'Value': '4'}],
-            StreamSpecification={'StreamEnabled': True, 'StreamViewType': 'KEYS_ONLY'},
-            KeySchema=[ { 'AttributeName': 'p', 'KeyType': 'HASH' }, ],
-            AttributeDefinitions=[ { 'AttributeName': 'p', 'AttributeType': 'S' } ]) as table:
-            pass
-    # Test attempting to add a stream to an existing table
-    with new_test_table(dynamodb,
-        Tags=[{'Key': 'experimental:initial_tablets', 'Value': '4'}],
-        KeySchema=[ { 'AttributeName': 'p', 'KeyType': 'HASH' }, ],
-        AttributeDefinitions=[ { 'AttributeName': 'p', 'AttributeType': 'S' } ]) as table:
-        with pytest.raises(ClientError, match='ValidationException.*tablets'):
-            table.update(StreamSpecification={'StreamEnabled': True, 'StreamViewType': 'KEYS_ONLY'});
-
 stream_types = [ 'OLD_IMAGE', 'NEW_IMAGE', 'KEYS_ONLY', 'NEW_AND_OLD_IMAGES']
 
 def disable_stream(dynamodbstreams, table):

test/alternator/test_tablets.py

@@ -0,0 +1,151 @@
+# Copyright 2024-present ScyllaDB
+#
+# SPDX-License-Identifier: LicenseRef-ScyllaDB-Source-Available-1.0
+
+# Tests for the Scylla-only "tablets" feature.
+#
+# Ideally, tablets are just an implementation detail (replacing the
+# old vnodes), that the DynamoDB API user would not even be aware
+# of. So there should be very few, if any, tests in this file.
+# However, temporarily - while the tablets feature is only partially
+# working and turned off by default (see issue #21989) - it is useful
+# to have here a few tests that clarify the situation and how to
+# override it. Most of these tests, or perhaps even this entire file,
+# will probably go away eventually.
+
+import pytest
+import boto3
+from botocore.exceptions import ClientError
+
+from .util import new_test_table
+
+# All tests in this file are scylla-only
+@pytest.fixture(scope="function", autouse=True)
+def all_tests_are_scylla_only(scylla_only):
+    pass
+
+# Utility function for checking if a given table is using tablets
+# or not. We rely on some knowledge of Alternator internals:
+# 1. For table with name X, Scylla creates a keyspace called alternator_X
+# 2. We can read a CQL system table using the ".scylla.alternator." prefix.
+def uses_tablets(dynamodb, table):
+    info = dynamodb.Table('.scylla.alternator.system_schema.scylla_keyspaces')
+    try:
+        response = info.query(
+            KeyConditions={'keyspace_name': {
+                    'AttributeValueList': ['alternator_'+table.name],
+                    'ComparisonOperator':  'EQ'}})
+    except dynamodb.meta.client.exceptions.ResourceNotFoundException:
+        # The internal Scylla table doesn't even exist, either this isn't
+        # Scylla or it's older Scylla and doesn't use tablets.
+        return False
+    if not 'Items' in response or not response['Items']:
+        return False
+    if 'initial_tablets' in response['Items'][0] and response['Items'][0]['initial_tablets']:
+        return True
+    return False
+
+# Right now, new Alternator tables are created *without* tablets.
+# This test should be changed if this default ever changes.
+def test_default_tablets(dynamodb):
+    schema = {
+        'KeySchema': [ { 'AttributeName': 'p', 'KeyType': 'HASH' } ],
+        'AttributeDefinitions': [ { 'AttributeName': 'p', 'AttributeType': 'S' }]}
+    with new_test_table(dynamodb, **schema) as table:
+        # Change this assertion if Alternator's default changes!
+        assert not uses_tablets(dynamodb, table)
+
+# Tests for the initial_tablets tag. Currently, it is considered
+# experimental, and named "experimental:initial_tablets", but perhaps
+# in the future it will graduate out of experimental status and
+# the prefix will be replaced by "system:".
+initial_tablets_tag = 'experimental:initial_tablets'
+
+# Check that a table created with a number as initial_tablets will use 
+# tablets. Different numbers have different meanings (0 asked to use
+# default number, any other number overrides the default) but they
+# all enable tablets.
+def test_initial_tablets_number(dynamodb):
+    for value in ['0', '4']:
+        schema = {
+            'Tags': [{'Key': initial_tablets_tag, 'Value': value}],
+            'KeySchema': [ { 'AttributeName': 'p', 'KeyType': 'HASH' } ],
+            'AttributeDefinitions': [ { 'AttributeName': 'p', 'AttributeType': 'S' }]}
+        with new_test_table(dynamodb, **schema) as table:
+            assert uses_tablets(dynamodb, table)
+
+# Check that a table created with a non-number (e.g., the string "none")
+# as initial_tablets, will not use tablets.
+def test_initial_tablets_number(dynamodb):
+    schema = {
+        'Tags': [{'Key': initial_tablets_tag, 'Value': 'none'}],
+        'KeySchema': [ { 'AttributeName': 'p', 'KeyType': 'HASH' } ],
+        'AttributeDefinitions': [ { 'AttributeName': 'p', 'AttributeType': 'S' }]}
+    with new_test_table(dynamodb, **schema) as table:
+        assert not uses_tablets(dynamodb, table)
+
+# Before Alternator TTL is supported with tablets (#16567), let's verify
+# that enabling TTL results in an orderly error. This test should be deleted
+# when #16567 is fixed.
+def test_ttl_enable_error_with_tablets(dynamodb):
+    with new_test_table(dynamodb,
+        Tags=[{'Key': initial_tablets_tag, 'Value': '4'}],
+        KeySchema=[ { 'AttributeName': 'p', 'KeyType': 'HASH' }, ],
+        AttributeDefinitions=[ { 'AttributeName': 'p', 'AttributeType': 'S' } ]) as table:
+        with pytest.raises(ClientError, match='ValidationException.*tablets'):
+            table.meta.client.update_time_to_live(TableName=table.name,
+                TimeToLiveSpecification={'AttributeName': 'expiration', 'Enabled': True})
+
+# Before Alternator Streams is supported with tablets (#16317), let's verify
+# that enabling Streams results in an orderly error. This test should be
+# deleted when #16317 is fixed.
+def test_streams_enable_error_with_tablets(dynamodb):
+    # Test attempting to create a table already with streams
+    with pytest.raises(ClientError, match='ValidationException.*tablets'):
+        with new_test_table(dynamodb,
+            Tags=[{'Key': initial_tablets_tag, 'Value': '4'}],
+            StreamSpecification={'StreamEnabled': True, 'StreamViewType': 'KEYS_ONLY'},
+            KeySchema=[ { 'AttributeName': 'p', 'KeyType': 'HASH' }, ],
+            AttributeDefinitions=[ { 'AttributeName': 'p', 'AttributeType': 'S' } ]) as table:
+            pass
+    # Test attempting to add a stream to an existing table
+    with new_test_table(dynamodb,
+        Tags=[{'Key': initial_tablets_tag, 'Value': '4'}],
+        KeySchema=[ { 'AttributeName': 'p', 'KeyType': 'HASH' }, ],
+        AttributeDefinitions=[ { 'AttributeName': 'p', 'AttributeType': 'S' } ]) as table:
+        with pytest.raises(ClientError, match='ValidationException.*tablets'):
+            table.update(StreamSpecification={'StreamEnabled': True, 'StreamViewType': 'KEYS_ONLY'});
+
+# Currently, LWT is not supported for tablets because of known bugs
+# (see #18068). We still allow creating an Alternator table with
+# tablets and using LWT for write isolation (always_use_lwt),
+# but the writes themselves will fail. When #18068 is fixed, this
+# test should be fixed to expect success - not failure.
+def test_alternator_tablets_and_lwt(dynamodb):
+    schema = {
+        'Tags': [
+            {'Key': initial_tablets_tag, 'Value': '0'},
+            {'Key': 'system:write_isolation', 'Value': 'always_use_lwt'}],
+        'KeySchema': [ { 'AttributeName': 'p', 'KeyType': 'HASH' } ],
+        'AttributeDefinitions': [ { 'AttributeName': 'p', 'AttributeType': 'S' }]}
+    with new_test_table(dynamodb, **schema) as table:
+        assert uses_tablets(dynamodb, table)
+        # This put_item should pass after #18068 is fixed:
+        with pytest.raises(ClientError, match="InternalServerError"):
+            table.put_item(Item={'p': 'hello'})
+            assert table.get_item(Key={'p': 'hello'})['Item'] == {'p': 'hello'}
+
+# An Alternator table created tablets and with a write isolation
+# mode that doesn't use LWT ("forbid_rmw") works normally, even
+# before #18068 is fixed.
+def test_alternator_tablets_without_lwt(dynamodb):
+    schema = {
+        'Tags': [
+            {'Key': initial_tablets_tag, 'Value': '0'},
+            {'Key': 'system:write_isolation', 'Value': 'forbid_rmw'}],
+        'KeySchema': [ { 'AttributeName': 'p', 'KeyType': 'HASH' } ],
+        'AttributeDefinitions': [ { 'AttributeName': 'p', 'AttributeType': 'S' }]}
+    with new_test_table(dynamodb, **schema) as table:
+        assert uses_tablets(dynamodb, table)
+        table.put_item(Item={'p': 'hello'})
+        assert table.get_item(Key={'p': 'hello'})['Item'] == {'p': 'hello'}

test/alternator/test_ttl.py

@@ -23,18 +23,6 @@ from test.alternator.util import new_test_table, random_string, full_query, uniq
 # using the following tags when creating each table below:
 TAGS = [{'Key': 'experimental:initial_tablets', 'Value': 'none'}]
 
-# Before Alternator TTL is supported with tablets (#16567), let's verify
-# that enabling TTL results in an orderly error. This test should be deleted
-# when #16567 is fixed.
-def test_ttl_enable_error_with_tablets(dynamodb, scylla_only):
-    with new_test_table(dynamodb,
-        Tags=[{'Key': 'experimental:initial_tablets', 'Value': '4'}],
-        KeySchema=[ { 'AttributeName': 'p', 'KeyType': 'HASH' }, ],
-        AttributeDefinitions=[ { 'AttributeName': 'p', 'AttributeType': 'S' } ]) as table:
-        with pytest.raises(ClientError, match='ValidationException.*tablets'):
-            table.meta.client.update_time_to_live(TableName=table.name,
-                TimeToLiveSpecification={'AttributeName': 'expiration', 'Enabled': True})
-
 # passes_or_raises() is similar to pytest.raises(), except that while raises()
 # expects a certain exception must happen, the new passes_or_raises()
 # expects the code to either pass (not raise), or if it throws, it must