Merge 'Introduce workload prioritization for service levels' from Piotr Dulikowski

This series introduces workload prioritization: an extension of the service levels feature which allows specifying "shares" per service level. The number of shares determines the priority of the user which has this service level attached (if multiple are attached then the one with the lowest shares wins).

Different service levels will be isolated in the following way:

- Each service level gets its own scheduling group with the number of shares (corresponding to the service level's number of shares), which controls the priority of the CPU and I/O used for user operations running on that service level.
- Each service level gets two reader concurrency semaphores, one for user reads and the other for read-before-write done for view updates.
- Each service level gets its own TCP connections for RPC to prevent priority inversion issues.

Because of the mandatory use of scheduling groups, which are a globally limited resource, the number of service levels is now limited to 7 user created service levels + 1 created by default that cannot be removed.

This feature has been previously only available in ScyllaDB Enterprise but has been made available for the source available ScyllaDB. The series was created by comparing the master branch with source-available-workbranch / enterprise branch and taking the workload prioritization related parts from the diff, then molding the resulting diff into a proper series. Some very minor changes were made such as fixing whitespace, removing unused or unnecessary code, adding some boilerplate (in api/) which was missing, but otherwise no major changes have been made.

No backport is required.

Closes scylladb/scylladb#22031

* github.com:scylladb/scylladb:
  tracing: record scheduling group in trace event record
  qos: un-shared-from-this standard_service_level_distributed_data_accessor
  alternator: execute under scheduling group for service level
  test.py: support multiple commands in prepare_cql in suite.yml
  docs: add documentation for workload prioritization
  docs/dev: describe workload prioritization features in service_levels
  test/auth_cluster: test workload prioritization in service level tests
  cqlpy/test_service_levels: add workload prioritization tests
  api: introduce service levels specific API
  api/cql_server_test: add information about scheduling group
  db/virtual_tables: add scheduling group column to system.clients
  test/boost: update service_level_controller_test for workload prio
  qos: include number of shares in DESCRIBE
  cql3/statements: update SL statements for workload prioritization
  transport/server: use scheduling group assigned to current user
  messaging_service: use separate set of connections per service levels
  replica/database: add reader concurrency semaphore groups
  qos: manage and assign scheduling groups to service levels
  qos: use the shares field in service level reads/writes
  qos: add shares to service_level_options
  qos: explicitly specify columns when querying service level tables
  db/system_distributed_keyspace: add shares column and upgrade code
  db/system_keyspace: adjust SL schema for workload prioritization
  gms: introduce WORKLOAD_PRIORITIZATION cluster feature
  build: increase the max number of scheduling groups
  qos: return correct error code when SL does not exist
This commit is contained in:
Avi Kivity
2025-01-02 20:05:36 +02:00
76 changed files with 3080 additions and 218 deletions

View File

@@ -159,6 +159,37 @@ If you don't know the name of the table, you can try a forbidden operation
and the AccessDeniedException error will contain the name of the table
that was lacking permissions.
## Workload Isolation
In DynamoDB read/write capacity of each table can be defined either to a fixed
value (provisioned mode) or to be adaptive (on-demand). On top of that requests
are also subject to per table and per account quotas.
Due to the nature of Alternator deployment the whole cluster is available to serve
user requests and underlying hardware can be utilized to its full capacity. When
there is a need to allow more resources to given workload on the expense of some competing
one we offer feature called **Workload Prioritization**.
To use this feature define service level with a fixed amount of shares
(higher value means proportionally more capacity) and attach it to a role
which then will be used to authorize requests. This can be currently done
only via CQL API, here is an example on how to do that:
```cql
CREATE ROLE alice WITH PASSWORD = 'abcd' AND LOGIN = true;
CREATE ROLE bob WITH PASSWORD = 'abcd' AND LOGIN = true;
CREATE SERVICE_LEVEL IF NOT EXISTS olap WITH SHARES = 100;
CREATE SERVICE_LEVEL IF NOT EXISTS oltp WITH SHARES = 1000;
ATTACH SERVICE_LEVEL olap TO alice;
ATTACH SERVICE_LEVEL oltp TO bob;
```
Note that `alternator_enforce_authorization` has to be enabled in Scylla configuration.
See [Authorization](##Authorization) section to learn more about roles and authorization.
See <https://enterprise.docs.scylladb.com/stable/using-scylla/workload-prioritization.html>
to read about **Workload Prioritization** in detail.
## Metrics
Scylla has an advanced and extensive monitoring framework for inspecting

View File

@@ -33,7 +33,8 @@ SELECT * FROM system.role_attributes WHERE role='r' and attribute_name='service
CREATE TABLE system_distributed.service_levels (
service_level text PRIMARY KEY,
timeout duration,
workload_type text)
workload_type text,
shares int);
```
The table is used to store and distribute the service levels configuration.
@@ -41,6 +42,7 @@ The table column names meanings are:
*service_level* - the name of the service level.
*timeout* - timeout for operations performed by users under this service level
*workload_type* - type of workload declared for this service level (NULL, interactive or batch)
*shares* - a number that represents this service level priority in relation to other service levels.
```
select * from system_distributed.service_levels ;
@@ -136,6 +138,35 @@ the conflicts are resolved as follows:
- `X` vs `NULL` -> `X`
- `batch` vs `interactive` -> `batch` - under the assumption that `batch` is safer, because it would not trigger load shedding as eagerly as `interactive`
So for example to create a service level that is twice more important than the default service
level (which has shares of 1000) one can run:
```
INSERT INTO system_distributed.service_level (service_level, shares) VALUES ('double_importance',2000);
```
## Service levels REST API
In a current state, Service Levels/Workload Prioritization has its own flaws, one of which is a requirement to restart connections to apply changes of users' service levels change.
Until we improve service levels controller to make the changes automatically, here is a REST API to ease to work of maintaining and managing service levels and connections.
A `tenant` (used below) is equal to scheduling group under which a connection is working.
### Switch tenants
`/service_levels/switch_tenants` endpoint triggers a tenant switch on all opened CQL connections on a single node without any interruption or their restart.
The response is returned immediately but the actual work might take up to tens of seconds.
### Inspecting current scheduling group of connections
`/service_levels/count_connections` endpoing is a tool to inspect status of all opened CQL connections. It returns a map with connections count per scheduling group, per user:
```
{'sl:default': {'cassandra': 3}, 'sl:sl1': {'test_user': 3}}
```
In fact, this endpoint is a wrapper which executes simple query on `system.clients` table and aggregates the result. The table has added `scheduling_group` column, so to inspect a particular connection, it can be directly looked up in `system.clients` table.
### Effective service level
Actual values of service level's options may come from different service levels, not only from the one user is assigned with. This can be achieved by assigning one role to another.

View File

@@ -14,6 +14,7 @@ This document highlights ScyllaDB's key data modeling features.
Counters </features/counters/>
Change Data Capture </features/cdc/index>
Workload Attributes </features/workload-attributes>
Workload Prioritization </features/workload-prioritization>
.. panel-box::
:title: ScyllaDB Features

View File

@@ -13,7 +13,7 @@ You can define a workload's attribute using the *service level* concept. The ser
attributes to users and roles. When a user logs into the system, all of the attributes attached to that user and to the roles
granted to that user are combined and become a set of workload attributes.
See `Service Level Management <https://enterprise.docs.scylladb.com/stable/using-scylla/workload-prioritization.html#workload-prioritization-workflow>`_ for more information about service levels.
See :ref:`Service Level Management <workload-priorization-service-level-management>` for more information about service levels.
Prerequisites
---------------
@@ -126,7 +126,7 @@ Available Workload Types
* - ``unspecified``
- A generic workload without any specific characteristics (default).
* - ``interactive``
- A workload sensitive to latency, expected to have high/unbounded concurrency, with dynamic characteristics. For example, a workload assigned to users clicking on a website and generating events with their clicks.
- A workload sensitive to latency, expected to have high/unbounded concurrency, with dynamic characteristics, :doc:`OLTP </features/workload-prioritization>`. For example, a workload assigned to users clicking on a website and generating events with their clicks.
* - ``batch``
- A workload for processing large amounts of data, not sensitive to latency, expected to have fixed concurrency. For example, a workload assigned to processing billions of historical sales records to generate statistics.
- A workload for processing large amounts of data, not sensitive to latency, expected to have fixed concurrency, :doc:`OLAP </features/workload-prioritization>`. For example, a workload assigned to processing billions of historical sales records to generate statistics.

View File

@@ -0,0 +1,448 @@
========================
Workload Prioritization
========================
:label-tip:`ScyllaDB Enterprise`
In a typical database there are numerous workloads running at the same time.
Each workload type dictates a different acceptable level of latency and throughput.
For example, consider the following two workloads:
* OLTP ( Online Transaction Processing) - backend database for your application
- High volume of requests
- Fast processing
- In essence - Latency sensitive
* OLAP (Online Analytical Processing ) - performs data analytics in the background
- High volume of data
- Slow queries
- In essence - Latency agnostic
Using Service Level CQL commands, database administrators (working on Scylla Enterprise) can set different workload prioritization levels (levels of service) for each workload without sacrificing latency or throughput.
By assigning each service level to the different roles within your organization, DBAs ensure that each role_ receives the level of service the role requires.
.. _`role` : /operating-scylla/security/rbac_usecase/
Prerequisites
=============
To create a level of service and assign it to a role, you need:
* An :doc:`authenticated </operating-scylla/security/runtime-authentication>` and :doc:`authorized </operating-scylla/security/enable-authorization>` user
* At least one :ref:`role created <create-role-statement>`.
Work by Example
---------------
To follow the examples in this document, create the roles `spark` and `web`. You can assign permissions to these roles later, if needed.
**Procedure**
Run the following:
.. code-block:: cql
CREATE ROLE Spark;
CREATE ROLE Web;
Workload Prioritization Workflow
================================
1. `Create a Service Level`_
2. `Assign a Service Level to a Role`_
.. _workload-priorization-service-level-management:
Service Level Management
========================
These commands set, list, and edit the level of service.
Create a Service Level
----------------------
When you create a service level, you allocate a percentage of resources to the service level. Remember to use the correct naming convention to name your service level. If you decide to use :doc:`Reserved Keywords </cql/reserved-keywords>`, enclose them in either single or double quotes (for example ``'primary'``).
**Syntax**
.. code-block:: none
CREATE SERVICE_LEVEL [IF NOT EXISTS] <service_level_name> [WITH SHARES = <shares_number>];
Where:
* ``service_level_name`` - Specifies the name of the service you're creating. This can be any string without spaces. The name should be meaningful, such as class names (silver, gold, diamond, platinum), or categories (OLAP or OLTP), etc.
* ``shares_number`` - The number of shares of the resources you're granting to the service level name. You can use any number within the range from 1 to 1000. **Default : 1000**
Example
.......
There are 3 service levels (OLAP, OLTP, Default) where: (the percentage of resources = (Assigned Shares / Total Shares) x 100). Total Shares in this case is the total of all allocated shares + the Default SLA (1000). The percentage of resources would be:
.. list-table::
:widths: 30 30 30
:header-rows: 1
* - Service Level Name
- Shares
- Percentage of Resources
* - OLAP
- 100
- 4%
* - OLTP
- 1000
- 48%
* - Default
- 1000
- 48%
* - Total
- 2100
- 100%
**Procedure**
1. To create these service levels, run the following CQL commands:
.. code-block:: cql
CREATE SERVICE_LEVEL IF NOT EXISTS OLAP WITH SHARES = 100;
CREATE SERVICE_LEVEL IF NOT EXISTS OLTP WITH SHARES = 1000;
2. Confirm the service level change reflects the new service level allocations:
.. code-block:: cql
LIST ALL SERVICE_LEVELS;
service_level | shares
--------------+-------
olap | 100
--------------+-------
oltp | 1000
(2 rows)
Change Resource Allocation for a Service Level
-----------------------------------------------
You can change resource allocation for a given service level. If you don't specify the number the shares, the default setting (1000) is used.
**Syntax**
.. code-block:: none
ALTER SERVICE_LEVEL <service_level_name>
WITH SHARES = <shares_number>;
Where:
* ``service_level_name`` - Specifies the name of the service level you created. See `Create a Service Level`_.
* ``shares_number`` - The number of shares in the CPU that you're granting to the service level name. You can use any number within the range from 1 to 1000. **Default : 1000**
.. warning::
Altering the SERVICE LEVEL does not affect active sessions (see `#12923 <https://github.com/scylladb/scylladb/issues/12923>`_).
To apply a new timeout to existing clients, execute a :doc:`rolling restart </operating-scylla/procedures/config-change/rolling-restart>` after the ALTER command.
Example
........
Analysts are complaining that they don't have enough resources. To increase the resources, you change the service level attributes for the OLAP service level.
**Procedure**
1. Run the following:
.. code-block:: cql
ALTER SERVICE_LEVEL OLAP WITH SHARES = 500;
2. Confirm the service level change reflects the new service level allocation:
.. code-block:: cql
LIST SERVICE_LEVEL OLAP;
service_level | shares
--------------+-------
olap | 500
(1 rows)
3. To change it back to the original setting (or to remain consistent for the examples that follow) change the shares amount back to the original.
.. code-block:: cql
ALTER SERVICE_LEVEL OLAP WITH SHARES = 100;
Display Specified Service Level Parameters
------------------------------------------
Lists the specified service level with its class parameters. If the service level is attached to a role it does not appear in this list.
**Syntax**
.. code-block:: none
LIST SERVICE_LEVEL <service_level_name>;
Where:
* ``service_level_name`` - Specifies the name of the service level you created. See `Create a Service Level`_.
Example
.......
In this example you list the service level parameters for OLTP.
**Procedure**
Run the following:
.. code-block:: cql
LIST SERVICE_LEVEL OLTP;
service_level | shares
--------------+-------
oltp | 1000
(1 rows)
Display All Service Levels and Parameters
-----------------------------------------
Lists all service levels with their class parameters. This list contains all service levels including those which are assigned to roles.
**Syntax**
.. code-block:: none
LIST ALL SERVICE_LEVELS;
Example
.......
In this example, you list all service levels and their parameters.
**Procedure**
Run the following:
.. code-block:: cql
LIST ALL SERVICE_LEVELS;
service_level | shares
---------------+--------
olap | 100
oltp | 1000
(2 rows)
Delete a Service Level
----------------------
Permanently removes the service level. Any role attached to this service level is automatically assigned to the Default SLA if there is no other service level attached to the role.
**Syntax**
.. code-block:: none
DROP SERVICE_LEVEL IF EXISTS <service_level_name>;
Where:
* ``service_level_name`` - Specifies the name of the service level you created. See `Create a Service Level`_.
* ``IF EXISTS`` - If the service level does not exist and IF EXISTS is not used an error is returned.
Example
.......
In this example you drop the OLTP service level.
**Procedure**
Run the following:
.. code-block:: cql
DROP SERVICE_LEVEL IF EXISTS OLTP;
Manage Roles with Service Levels
================================
Once you have created roles and service levels you can attach and remove the service levels from the roles and list which roles are attached to which service levels.
Assign a Service Level to a Role
--------------------------------
If you have created a role and a service level, you can attach the service level to the role.
.. note:: A role can only be assigned **one** service level. However, the same service level can be attached to many roles. If a role inherits a service level from another role, the highest level of service from all the roles wins.
**Syntax**
.. code-block:: none
ATTACH SERVICE_LEVEL <service_level_name> TO <role_name>;
Where:
* ``service_level_name`` - Specifies the name of the service level you created. See `Create a Service Level`_.
* ``role_name`` - Specifies the role that you want to use the service level on. This is the role you created with :ref:`create role <create-role-statement>`.
.. note:: Any role which does not have an SLA attached to it, receives the default SLA.
Example
.......
Continuing from the example in `Create a Service Level`_, you can attach the service levels that you created to different roles in your organization as follows:
.. list-table::
:widths: 50 50
:header-rows: 1
* - Service Level Name
- Role Name
* - OLAP
- Spark
* - OLTP
- Web
**Procedure**
To assign these service levels to the roles, run the following CQL commands:
.. code-block:: cql
ATTACH SERVICE_LEVEL OLAP TO Spark;
ATTACH SERVICE_LEVEL OLTP TO Web;
List All Attached Service Levels for All Roles
----------------------------------------------
Lists all directly attached service levels for all roles. This does not include any service level which the role inherits from other roles.
**Syntax**
.. code-block:: none
LIST ALL ATTACHED SERVICE_LEVELS;
Example
.......
In this example you list all service levels attached to any role.
**Procedure**
Run the following:
.. code-block:: cql
LIST ALL ATTACHED SERVICE_LEVELS;
role | service_level
-------+---------------
spark | olap
-------+---------------
web | oltp
(2 rows)
List the Roles Assigned to a Specific Service Level
----------------------------------------------------
Lists all roles directly attached to a service level. This does not include any service level which the role inherits from other roles.
**Syntax**
.. code-block:: none
LIST ATTACHED SERVICE_LEVEL OF <role_name>;
Where:
* ``role_name`` - Specifies the role that you want to use the service level on. This is the role you created with :ref:`create role <create-role-statement>`.
Example
.......
In this example, you list all of Roles which are assigned to the OLAP Service Level.
**Procedure**
Run the following:
.. code-block:: cql
LIST ATTACHED SERVICE_LEVEL OF Spark;
role | service_level
-------+---------------
spark | olap
(1 rows)
Remove a Service Level from a Role
----------------------------------
Removes a service level from a specified role. Once the service level is removed from a role, if there are other service levels attached to roles which that role inherits, the service level in the hierarchy with the most amount of shares wins.
**Syntax**
.. code-block:: none
DETACH SERVICE_LEVEL FROM <role_name>;
Where:
* ``role_name`` - Specifies the role that you want to use the service level on. This is the role you created with :ref:`create role <create-role-statement>`.
Example
.......
In this example, you re-assign the Spark to a different level of service by detaching it from one level of service and attaching it to another.
**Procedure**
Run the following:
.. code-block:: cql
DETACH SERVICE_LEVEL FROM Spark;
At this point, the Spark role receives the Default SLA, until it is assigned another service level. You assign a new service level to this role using `Assign a Service Level to a Role`_.
Using Workload Prioritization with your Application
===================================================
In order for workload prioritization to take effect, application users need to be assigned to a relevant role. In addition, each role you create needs to be assigned to a specific Service Level. Any user that signs into the application without a role is automatically assigned the `Default` service level. This is always be the case with users who sign in anonymously.
Limits
======
Scylla Enterprise is limited to 8 service levels, including the default one; this means you can create up to 7 service levels.
Additional References
=====================
`OLAP or OLTP? Why Not Both? <https://www.youtube.com/watch?v=GhmgwN6ZraI>`_ Session by Glauber Costa from Scylla Summit 2018
`Scylla University: Workload Prioritization lesson <https://university.scylladb.com/courses/scylla-operations/lessons/workload-prioritization/>`_ - The lesson covers:
* The evolving requirements for operational (OLTP) and analytics (OLAP) workloads in the modern datacenter
* How Scylla provides built-in control over workload priority and makes it easy for administrators to configure workload priorities
* The impact of minimizing integrations and maintenance tasks, while also shrinking the datacenter footprint and maximizing utilization
* Test results of how it performs in real-world settings

View File

@@ -22,7 +22,7 @@ In the same manner, should someone leave the organization, all you would have to
Should someone change positions at the company, just assign the new employee to the new role and revoke roles no longer required for the new position.
To build an RBAC environment, you need to create the roles and their associated permissions and then assign or grant the roles to the individual users. Roles inherit the permissions of any other roles that they are granted. The hierarchy of roles can be either simple or extremely complex. This gives great flexibility to database administrators, where they can create specific permission conditions without incurring a huge administrative burden.
In addition to standard roles, `ScyllaDB Enterprise <https://enterprise.docs.scylladb.com/>`_ users can implement `Workload Prioritization <https://enterprise.docs.scylladb.com/stable/using-scylla/workload-prioritization.html>`_, which allows you to attach roles to Service Levels, thus granting resources to roles as the role demands.
In addition to standard roles, ScyllaDB Enterprise users can implement :doc:`Workload Prioritization </features/workload-prioritization/>`, which allows you to attach roles to Service Levels, thus granting resources to roles as the role demands.
.. _rbac-usecase-grant-roles-and-permissions:
@@ -213,4 +213,4 @@ Additional References
* :doc:`Authorization</operating-scylla/security/authorization/>`
* :doc:`CQLSh the CQL shell</cql/cqlsh>`
* `Workload Prioritization <https://enterprise.docs.scylladb.com/stable/using-scylla/workload-prioritization.html>`_ - to attach a service level to a role. Only available in `ScyllaDB Enterprise <https://enterprise.docs.scylladb.com/>`_.
* :doc:`Workload Prioritization </features/workload-prioritization/>` - to attach a service level to a role