The helper consists of three parts -- prepare the deletion log, unlink
sstables and drop the deletion log. For testing the first part is needed
as a separate step, so here's this split.
It renders two nested async contexts, but it will change soon.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The ".github/CODEOWNERS" is used by github to recommend reviewers for
pull requests depending on the directories touched in the pull request.
Github ignores entries on that file who are not **maintainers**. Since
Jan is no longer a Scylla maintainer, I remove his entries in the list.
Additionally, I am removing *myself* from *some* of the directories.
For many years, it was an (unwritten) policy that experienced Scylla
developers are expected to help in reviewing pieces of the code they
are familiar with - even if they no longer work on that code today.
But as ScyllaDB the company grew, this is no longer true; The policy
is now that experienced developers are requested review only code in
their own or their team's area of responsibility - experienced developers
should help review *designs* of other parts, but not the actual code.
For this reason I'm removing my name from various directories.
I can still help review such code if asked specifically - but I will no
longer be the "default" reviewer for such code.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closesscylladb/scylladb#16239
Add `LIST EFFECTIVE SERVICE LEVEL` statement to be able to display from which service level come which service level options.
Example:
There are 2 roles: role1 and role2. Role1 is assigned with sl1 (timeout = 2s, workload_type = interactive) and role2 is assigned with sl2 (timeout = 10s, workload_type = batch).
Then, if we grant role1 to role2, the user with role2 will have 2s timeout (from sl1) and batch workload type (from sl2).
```
> LIST EFFECTIVE SERVICE LEVEL OF role2;
service_level_option | effective_service_level | value
----------------------+-------------------------+-------------
workload_type | sl2 | batch
timeout | sl1 | 2s
```
Fixes: https://github.com/scylladb/scylladb/issues/15604Closesscylladb/scylladb#14431
* github.com:scylladb/scylladb:
cql-pytest: add `LIST EFFECTIVE SERVICE LEVEL OF` test
docs: add `LIST EFFECTIVE SERVICE LEVEL` statement docs
cql3:statements: add `LIST EFFECTIVE SERVICE LEVEL` statement
service:qos: add option to include effective names to SLO
Storage service uses group0 internally, but group0 is create long after
storage service is initialized and passed to it using ss::set_group0()
function. But what it means is that during shutdown group0 is destroyed
before ss::stop() is called and thus storage service is left with a
dangling reference. Fix it by introducing a function that cancels all
group0 operations and waits for background fibers to complete. For that
we need separate abort source for group0 operation which the patch also
introduces.
We move the remaining tests in topology_raft_disabled to
topology_custom. We choose topology_custom because these tests
cannot use consistent topology changes.
We need to modify these tests a bit because we cannot pass
RandomTables to a test case function if the initial cluster size
equals 0. RandomTables.__init__ requires manager.cql to be present.
We move all used util functions from topology_raft_disabled to
topology before we remove topology_raft_disabled. After this
change, util.py in topology will be the single util file for all
topology tests.
Some util functions in topology_raft_disabled aren't used anymore.
We don't move such functions and remove them instead.
Soon, we will make consistent cluster management mandatory on
master. Before this, we have to change all tests in the
topology_raft_disabled suite so that they do not depend on the
consistent_cluster_management=false config.
Adapting test_raft_upgrade_majority_loss is simple. We only have
to get rid of the initial upgrade. This initial upgrade didn't
test anything. Every test in topology_raft_disabled had to do it
at the beginning because of consistent_cluster_management=false.
Adapting test_raft_upgrade_basic and test_raft_upgrade_stuck is
more difficult. It requires changing the initial upgrade to
clearing Raft data in RECOVERY mode on all servers and restarting
them. Then, the servers will run the same upgrade procedure as
before.
After changing the tests, we also update their names appropriately.
test_raft_upgrade_stuck becomes a bit slower, so we remove the
comment about running time. Also, one TODO was fixed in the process
of rewriting the test. This fix forced us to skip the test in the
release mode since we cannot update the list of error injections
through manager.server_update_config in this mode.
After making consistent cluster management mandatory on master,
schema commitlog will also become mandatory because
consistent_cluster_management: True,
force_schema_commit_log: False
is considered a bad configuration. These changes will make
test_raft_upgrade_no_schema unimplementable in the Scylla repo, so
we remove it.
If we want to keep this test, we must rewrite it as an upgrade
dtest.
under most circumstances, we don't care the ordering of the sstable
identifiers, as they are just identifiers. so, as long as they can be
compared, we are good. but we have tests with expect that the sstables
can be ordered by the time they are created. for instance,
sstable_run_based_compaction_test has this expectaion.
before this change, we compare two UUID-based generations by its
(MSB, LSB) lexicographically. but UUID v1 put the lower bits of
the timestamp at the higher bits of MSB, so the ordering of the
"time" in timeuuid is not preserved when comparing the UUID-based
generations. this breaks the test of sstable_run_based_compaction_test,
which feeds the sstables to be compacted in a set, and the set is
ordered with the generation of the sstables.
after this change, we consider the UUID-based generation as
a timeuuid when comparing them.
Fixes#16215
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#16238
Allow to include `slo_effective_names` in `service_level_options`
to be able to determine from which service level the specific option value comes from.
When scanning our latest docker image using `trivy` (command: `trivy
image docker.io/scylladb/scylla-nightly:latest`), it shows we have OS
packages which are out of date.
Also removing `openssh-server` and `openssh-client` since we don't use
it for our docker images
Fixes: https://github.com/scylladb/scylladb/issues/16222Closesscylladb/scylladb#16224
Said commands print errors as they validate the sstables. Currently this
intermingles with the regular JSON output of these commands, resulting
in ugly and confusing output.
This is not a problem for scripted use, as logs go to stderr while the
JSON go to stdout, but it is a problem for human users.
Solve this by outputting the JSON into a std::stringstream and printing
it in one go at the very end. This means JSON is accumulated in a memory
buffer, but these commands don't output a lot of JSON, so this shouldn't
be a problem.
Closesscylladb/scylladb#16216
Current mechanism to deprecate config options is implemented in a hacky
way in `main.cpp` and doesn't account for existing
`db::config/boost::po` API controlling lifetime of config options, hence
it's being replaced in this PR by adding yet another `value_status`
enumerator: `Deprecated`, so that deprecation of config options is
controlled in one place in `config.cc`,i.e. when specifying config options.
Motivation: https://docs.google.com/document/d/18urPG7qeb7z7WPpMYI2V_lCOkM5YGKsEU78SDJmt8bM/edit?usp=sharing
With this change, if a `Deprecated` config option is specified as
1. a command line parameter, scylla will run and log:
```
WARN 2023-11-25 23:37:22,623 [shard 0:main] init - background-writer-scheduling-quota option ignored (deprecated)
```
(Previously it was only a message printed to standard output, not a
scylla log of warn level).
2. an option in `scylla.yaml`, scylla will run and log:
```
WARN 2023-11-27 23:55:13,534 [shard 0:main] init - Option is deprecated : background_writer_scheduling_quota
```
Fixes#15887
Incorporates dropped https://github.com/scylladb/scylladb/pull/15928Closesscylladb/scylladb#16184
This miniset, completes the prerequisites for enabling commitlog hard limit on by default.
Namely, start flushing and evacuating segments halfway to the limit in order to never hit it under normal circumstances.
It is worth mentioning that hitting the limit is an exceptional condition which it's root cause need to be resolved, however,
once we do hit the limit, the performance impact that is inflicted as a result of this enforcement is irrelevant.
Tests: unit tests.
LWT write test (#9331)
A whitebox testing has been performed by @wmitros , the test aimed at putting as much pressure as possible on the commitlog segments by using a write pattern that rewrites the partitions in the memtable keeping it at ~85% occupancy so the dirty memory manager will not kick in. The test compared 3 configurations:
1. The default configuration
2. Hard limit on (without changing the flush threshold)
3. the changes in this PR applied.
The last exhibited the "best" behavior in terms of metrics, the graphs were the flattest and less jaggy from the others.
Closesscylladb/scylladb#10974
* github.com:scylladb/scylladb:
commitlog: enforce commitlog size hard limit by default
commitlog: set flush threshold to half of the limit size
commitlog: unfold flush threshold assignment
This series adds handling for more failures during a topology operation
(we already handle a failure during streaming). Here we add handling of
tablet draining errors by aborting the operation and handling of errors
after streaming where an operation cannot be aborted any longer. If the
error happens when rollback is no longer possible we wait for ring delay
and proceed to the next step. Each individual patch that adds the sleep
has an explanation what the consequences of the patch are.
* 'gleb/topology-coordinator-failures' of github.com:scylladb/scylla-dev:
test: add test to check errro handling during tablet draining
test: fix test_topology_streaming_failure test to not grep the whole file
storage_service: add error injection into the tablet migration code
storage_service: topology coordinator: rollback on handle_tablet_migration failure during tablet_draining stage
storage_service: topology coordinator: do not retry the metadata barrier forever in write_both_read_new state
storage_service: topology coordinator: do not retry the metadata barrier forever in left_token_ring state
storage_service: topology coordinator: return a node that is being removed from get_excluded_nodes
storage_service: topology_coordinator: use new rollback_to_normal state in the rollback procedure
storage_service: topology coordinator: add rollback_to_normal node state
storage_service: topology coordinator: put fence version into the raft state
storage_service: topology coordinator: do fencing even if draining failed
TOC file is read and parsed in several places in the code. All do it differently, and it's worth generalizing this place.
To make it happen also fix the S3 readable_file so that it could be used inside file_input_stream.
Closesscylladb/scylladb#16175
* github.com:scylladb/scylladb:
sstable: Generalize toc file read and parse
s3/client: Don't GET object contents on out-of-bound reads
s3/client: Cache stats on readable_file
This situation before this patch is that when tablets are enabled for
a keyspace, we can create a materialized view but later any write to
the base table fails with an on_internal_error(), saying that:
"Tried to obtain per-keyspace effective replication map of test
but it's per-table."
Indeed, with tablets, the replication is different for each table - it's
not the same for the entire keyspace.
So this patch changes the view update code to take the replication
map from the specific base table, not the keyspace.
This is good enough to get materialized-views reads and writes working
in a simple single-node case, as the included test demonstrates (the
test fails with on_internal_error() before this patch, and passes
afterwards).
But this fix is not perfect - the base-view pairing code really needs
to consider not only the base table's replication map, but also the
view table's replication map - as those can be different. We'll fix
this remaining problem as a followup in a separate patch - it will
require a substantially more elaborate test to reproduce the need
for the different mapping and to verify that fix.
Fixes#16209.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closesscylladb/scylladb#16211
* seastar 830ce8673...55a821524 (34):
> Revert "reactor/scheduling_group: Handle at_destroy queue special in init_new_scheduling_group_key etc"
> epoll: Avoid spinning on aborted connections
Fixes#12774Fixes#7753Fixes#13337
> Merge 'Sanitize test-only reactor facilities' from Pavel Emelyanov
> test/unit: fix fmt version check
> reactor/scheduling_group: Handle at_destroy queue special in init_new_scheduling_group_key etc
> build: add spaces before () and after commands
> reactor: use zero-initialization to initialize io_uring_params
> Merge 'build: do not return a non-false condition if the option is off ' from Kefu Chai
> memory: do not use variable length array
> build: use tri_state_option() to link against Sanitizers
> build: do not define SEASTAR_TYPE_ERASE_MORE on all builds
> Revert "shared_future: make available() immediate after set_value()"
> test_runner: do not throw when seastar.app fails to start
> Merge 'Address issue where Seastar faults in toeplitz hash when reassembling fragment' from John Hester
> defer, closeable: do not use [[nodiscard(str)]]
> Merge 'build: generate config-specific rules using generator expressions' from Kefu Chai
> treewide: use *_v and *_t for better readability
> build: use different names for .pc files for each build mode
> perftune.py: skip discovering IRQs for iSCSI disks
> io-tester: explicit use uint64_t for boost::irange(...)
> gate: correct the typo in doxygen comment
> shared_future: make available() immediate after set_value()
> smp: drop unused templates
> include fmt/ostream.h to make headers self-sufficient
> Support ccache in ./configure.py
> rpc_tester: Disable -Wuninitialized when including boost.accumulators
> file: construct directory_entry with aggregated ctor
> file: s/ino64_t/ino_t/, s/off64_t/off_t/
> sstring_test: include fmt/std.h only if fmtlib >= 10.0.0
> file: do not include coroutine headers if coroutine is disabled
> fair_queue::unregister_priority_class:fix assertion
> Merge 'Generalize `net::udp_channel` into `net::datagram_channel`' from Michał Sala
> Merge 'Add file::list_directory() that co_yields entries' from Pavel Emelyanov
> http/file_handler: remove unnecessary cast
Closesscylladb/scylladb#16201
There are several places where TOC file is parsed into a vector of
components -- sstable::read_toc(), remove_by_toc_name() and
remove_by_registry_entry(). All three deserve some generalization.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
If S3 readable file is used inside file input stream, the latter may
call its read methods with position that is above file size. In that
case server replies with generic http error and the fact that the range
was invalid is encoded into reply body's xml.
That's not great to catch this via wrong reply status exception and xml
parsing all the more so we can know that the read is out-of-bound in
advance.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
S3-based sstables components are immutable, so every time stat is called
there's no need to ping server again.
But the main intention of this patch is to provide stats for read calls
in the next patch.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Fixes#16207
commitlog::delete_segments deletes (or recycles) segments replayed.
The actual file size here is added to footprint so actual delete then
can determine iff things should be recycled or removed.
However, we build a pending delete list of named_files, and the files
we added did not have size set. Bad. Actual deletion then treated files
as zero-byte sized, i.e. footprint calculations borked.
Simple fix is just filling in the size of the objects when addind.
Added unit test for the problem.
Closesscylladb/scylladb#16210
Major compaction already flushes each table to make
sure it considers any mutations that are present in the
memtable for the purpose of tombstone purging.
See 64ec1c6ec6
However, tombstone purging may be inhibited by data
in commitlog segments based on `gc_time_min` in the
`tombstone_gc_state` (See f42eb4d1ce).
Flushing all sstables in the database release
all references to commitlog segments and there
it maximizes the potential for tombstone purging,
which is typically the reason for running major compaction.
However, flushing all tables too frequently might
result in tiny sstables. Since when flushing all
keyspaces using `nodetool flush` the `force_keyspace_compaction`
api is invoked for keyspace successively, we need a mechanism
to prevent too frequent flushes by major compaction.
Hence a `compaction_flush_all_tables_before_major_seconds` interval
configuration option is added (defaults to 24 hours).
In the case that not all tables are flushed prior
to major compaction, we revert to the old behavior of
flushing each table in the keyspace before major-compacting it.
Fixesscylladb/scylladb#15777Closesscylladb/scylladb#15820
* github.com:scylladb/scylladb:
docs: nodetool: flush: enrich examples
docs: nodetool: compact: fix example
api: add /storage_service/compact
api: add /storage_service/flush
compaction_manager: flush_all_tables before major compaction
database: add flush_all_tables
api: compaction: add flush_memtables option
test/nodetool: jmx: fix path to scripts/scylla-jmx
scylla-nodetool, docs: improve optional params documentation
our CI builds "dev-headers" as a gating check. but the target names
generated by CMake's Ninja Multi-Config generator does not follow
this naming convention. we could have headers:Dev, but still, it's
different from what we are using, before completely switching to
CMake, let's keep this backward compatibility by adding a target
with the same name.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
as part of the efforts to migrate to the CMake-based building system,
this change enables us to `configure.py` to optionally create
`build.ninja` with CMake.
in this change, we add a new option named `--use-cmake` to
`configure.py` so we can create `build.ninja`. please note,
instead of using the "Ninja" generator used by Seastar's
`configure.py` script, we use "Ninja Multi-Config" generator
along with `CMAKE_CROSS_CONFIGS` setting in this project.
so that we can generate a `build.ninja` which is capable of
building the same artifacts with multiple configuration.
Fixes#15379
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Before this patch, trying to create a materialized view when tablets
are enabled for a keyspace results in a failure: "Tablet map not found
for table <uuid>", with uuid referring to the new view.
When a table schema is created, the handler on_before_create_column_family()
is called - and this function creates the tablet map for the new table.
The bug was that we forgot to do the same when creating a materialized
view - which also a bona-fide table.
In this patch we call on_before_create_column_family() also when
creating the materialized view. I decided *not* to create a new
callback (e.g., on_before_create_view()) and rather call the existing
on_before_create_column_family() callback - after all, a view is
a column family too.
This patch also includes a test for this issue, which fails to create
the view before this patch, and passes with the patch. The test is
in the test/topology_experimental_raft suite, which runs Scylla with
the tablets experimental feature, and will also allow me to create
tests that need multiple nodes. However, the first test added here
only needs a single node to reproduce the bug and validate its fix.
Fixes#16194.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closesscylladb/scylladb#16205
When a view schema is changed, the schema change command also includes
mutations for the corresponding base table; these mutations don't modify
the base schema but are included in case if the receiver of view
mutations somehow didn't receive base mutations yet (this may in theory
happen outside Raft mode).
There are situations where the schema change command contains both
mutations that describe the current state of the base table -- included
by a view update, as explained above -- and mutations that want to
modify the base table. Such situation arises, for example, when we
update a user-defined type which is referenced by both a view and its
corresponding base table. This triggers a schema change of the view,
which generates mutations to modify the view and includes mutations of
the current base schema, and at the same time it triggers a schema
change of the base, which generates mutations to modify the base.
These two sets of mutations are conflicting with each other. One set
wants to preserve the current state of the base table while the other
wants to modify it. And the two sets of mutations are generated using
the same timestamp, which means that conflict resolution between them is
made on a per-mutation-cell basis, comparing the values in each cell and
taking the "larger" one (meaning of "larger" depends on the type of each
cell).
Fortunately, this conflict is currently benign -- or at least there is
no known situation where it causes problems.
Unfortunately, it started causing problems when I attempted to implement
group 0 schema versioning (PR scylladb/scylladb#15331), where instead of
calculating table versions as hashes of schema mutations, we would send
versions as part of schema change command. These versions would be
stored inside the `system_schema.scylla_tables` table, `version` column,
and sent as part of schema change mutations.
And then the conflict showed. One set of mutations wanted to preserve
the old value of `version` column while the other wanted to update it.
It turned out that sometimes the old `version` prevailed, because the
`version` column in `system_schema.scylla_tables` uses UUID-based
comparison (not timeuuid-based comparison). This manifested as issue
scylladb/scylladb#15530.
To prevent this, the idea in this commit is simple: when generating
mutations for the base table as part of corresponding view update, do
not use the provided timestamp directly -- instead, decrement it by one.
This way, if the schema change command contains mutations that want to
modify the base table, these modifying mutations will win all conflicts
based on the timestamp alone (they are using the same provided
timestamp, but not decremented).
One could argue that the choice of this timestamp is anyway arbitrary.
The original purpose of including base mutations during view update was
to ensure that a node which somehow missed the base mutations, gets them
when applying the view. But in that case, the "most correct" solution
should have been to use the *original* base mutations -- i.e. the ones
that we have on disk -- instead of generating new mutations for the base
with a refreshed timestamp. The base mutations that we have on disk have
smaller timestamps already (since these mutations are from the past,
when the base was last modified or created), so the conflict would also
not happen in this case.
But that solution would require doing a disk read, and we can avoid the
read while still fixing the conflict by using an intermediate solution:
regenerating the mutations but with `timestamp - 1`.
Ref: scylladb/scylladb#15530Closesscylladb/scylladb#16139
It looks like `nodetool compact standard1` is meant
to show how to compact a specified table, not a keyspace.
Note that the previous example like is for a keyspace.
So fix the table compaction example to:
`nodetool compact keyspace1 standard1`
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
For major compacting all tables in the database.
The advantage of this api is that `commitlog->force_new_active_segment`
happens only once in `database::flush_all_tables` rather than
once per keyspace (when `nodetool compact` translates to
a sequence of `/storage_service/keyspace_compaction` calls).
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
For flushing all tables in the database.
The advantage of this api is that `commitlog->force_new_active_segment`
happens only once in `database::flush_all_tables` rather than
once per keyspace (when `nodetool flush` translates to
a sequence of `/storage_service/keyspace_flush` calls).
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Major compaction already flushes each table to make
sure it considers any mutations that are present in the
memtable for the purpose of tombstone purging.
See 64ec1c6ec6
However, tombstone purging may be inhibited by data
in commitlog segments based on `gc_time_min` in the
`tombstone_gc_state` (See f42eb4d1ce).
Flushing all sstables in the database release
all references to commitlog segments and there
it maximizes the potential for tombstone purging,
which is typically the reason for running major compaction.
However, flushing all tables too frequently might
result in tiny sstables. Since when flushing all
keyspaces using `nodetool flush` the `force_keyspace_compaction`
api is invoked for keyspace successively, we need a mechanism
to prevent too frequent flushes by major compaction.
Hence a `compaction_flush_all_tables_before_major_seconds` interval
configuration option is added (defaults to 24 hours).
In the case that not all tables are flushed prior
to major compaction, we revert to the old behavior of
flushing each table in the keyspace before major-compacting it.
Fixesscylladb/scylladb#15777
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Flushes all tables after forcing force_new_active_segment
of the commitlog to make sure all commitlog segments can
get recycled.
Otherwise, due to "false sharing", rarely-written tables
might inhibit recycling of the commitlog segments they reference.
After f42eb4d1ce,
that won't allow compaction to purge some tombstones based on
the min_gc_time.
To be used in the next patch by major compaction.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
When flushing is done externally, e.g. by running
`nodetool flush` prior to `nodetool compact`,
flush_memtables=false can be passed to skip flushing
of tables right before they are major-compacted.
This is useful to prevent creation of small sstables
due to excessive memtable flushing.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
The current implementation makes no sense.
Like `nodetool_path`, base the default `jmx_path`
on the assumption that the test is run using, e.g.
```
(cd test/nodetool; pytest --nodetool=cassandra test_compact.py)
```
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Document the behavior if no keyspace is specified
or no table(s) are specified for a given keyspace.
Fixesscylladb/scylladb#16032
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
This commit adds information on how to enable
object storage for a keyspace.
The "Keyspace storage options" section already
existed in the doc, but it was not valid as
the support was only added in version 5.4
The scope of this commit:
- Update the "Keyspace storage options" section.
- Add the information about object storage support
to the Data Definition> CREATE KEYSPACE section
* Marked as "Experimental".
* Excluded from the Enterprise docs with the
.. only:: opensource directive.
This commit must be backported to branch-5.4,
as support for object storage was added
in version 5.4.
Closesscylladb/scylladb#16081
Nowadays if memtable gets flushed into misconfigured S3 storage, the flush fails and aborts the whole scylla process. That's not very elegant. First, because upon restart garbage collecting non-sealed sstables would fail again. Second, because re-configuring an endpoint can be done runtime, scylla re-reads this config upon HUP signal.
Flushing memtable restarts when seeing ENOSPC/EDQUOT errors from on-disk sstables. This PR extends this to handle misconfigured S3 endpoints as well.
fixes: #13745Closesscylladb/scylladb#15635
* github.com:scylladb/scylladb:
test: Add object_store test to validate config reloading works
test: Add config update facility to test cluster
test: Make S3_Server export config file as pathlib.Path
config: Make object storage config updateable_value_source
memtable: Extend list of checking codes
sstables/storage/s3: Fix missing TOC status check
s3/client: Map http exceptions into storage_io_error
exceptions: Extend storage_io_error construction options
Compaction tasks which do not have a parent are abortable
through task manager. Their children are aborted recursively.
Compaction tasks of the lowest level are aborted using existing
compaction task executors stopping mechanism.
Closesscylladb/scylladb#16177
* github.com:scylladb/scylladb:
test: test abort of compaction task that isn't started yet
test: test running compaction task abort
tasks: fail if a task was aborted
compaction: abort task manager compaction tasks
