Commit Graph

40113 Commits

Author SHA1 Message Date
Botond Dénes
d2a88cd8de Merge 'Typos: fix typos in code' from Yaniv Kaul
Fixes some more typos as found by codespell run on the code. In this commit, there are more user-visible errors.

Refs: https://github.com/scylladb/scylladb/issues/16255

Closes scylladb/scylladb#16289

* github.com:scylladb/scylladb:
  Update unified/build_unified.sh
  Update main.cc
  Update dist/common/scripts/scylla-housekeeping
  Typos: fix typos in code
2023-12-06 07:36:41 +02:00
Avi Kivity
12f160045b Merge 'Get rid of fb_utilities' from Benny Halevy
utils::fb_utilities is a global in-memory registry for storing and retrieving broadcast_address and broadcat_rpc_address.
As part of the effort to get rid of all global state, this series gets rid of fb_utilities.
This will eventually allow e.g. cql_test_env to instantiate multiple scylla server nodes, each serving on its own address.

Closes scylladb/scylladb#16250

* github.com:scylladb/scylladb:
  treewide: get rid of now unused fb_utilities
  tracing: use locator::topology rather than fb_utilities
  streaming: use locator::topology rather than fb_utilities
  raft: use locator::topology/messaging rather than fb_utilities
  storage_service: use locator::topology rather than fb_utilities
  storage_proxy: use locator::topology rather than fb_utilities
  service_level_controller: use locator::topology rather than fb_utilities
  misc_services: use locator::topology rather than fb_utilities
  migration_manager: use messaging rather than fb_utilities
  forward_service: use messaging rather than fb_utilities
  messaging_service: accept broadcast_addr in config rather than via fb_utilities
  messaging_service: move listen_address and port getters inline
  test: manual: modernize message test
  table: use gossiper rather than fb_utilities
  repair: use locator::topology rather than fb_utilities
  dht/range_streamer: use locator::topology rather than fb_utilities
  db/view: use locator::topology rather than fb_utilities
  database: use locator::topology rather than fb_utilities
  db/system_keyspace: use topology via db rather than fb_utilities
  db/system_keyspace: save_local_info: get broadcast addresses from caller
  db/hints/manager: use locator::topology rather than fb_utilities
  db/consistency_level: use locator::topology rather than fb_utilities
  api: use locator::topology rather than fb_utilities
  alternator: ttl: use locator::topology rather than fb_utilities
  gossiper: use locator::topology rather than fb_utilities
  gossiper: add get_this_endpoint_state_ptr
  test: lib: cql_test_env: pass broadcast_address in cql_test_config
  init: get_seeds_from_db_config: accept broadcast_address
  locator: replication strategies: use locator::topology rather than fb_utilities
  locator: topology: add helpers to retrieve this host_id and address
  snitch: pass broadcast_address in snitch_config
  snitch: add optional get_broadcast_address method
  locator: ec2_multi_region_snitch: keep local public address as member
  ec2_multi_region_snitch: reindent load_config
  ec2_multi_region_snitch: coroutinize load_config
  ec2_snitch: reindent load_config
  ec2_snitch: coroutinize load_config
  thrift: thrift_validation: use std::numeric_limits rather than fb_utilities
2023-12-05 19:40:14 +02:00
Eliran Sinvani
d1aaca893c install-dependencies.sh: Complete the pip install logic
install-dependencies.sh includes a list of pip packages that the build
environment requires.
This functionality was added in
729d0feef0, however, the actual use of the
list is missing and instead the `pip install` commands are hard coded
into the logic.

This change complete the transition to pip-packages list.
It includes also modifying the `pip_packages` array to include a
constrain (if needed) for every package.

Fixes #16269

Signed-off-by: Eliran Sinvani <eliransin@scylladb.com>

Closes scylladb/scylladb#16282
2023-12-05 16:35:31 +02:00
Benny Halevy
0bcce35abd treewide: get rid of now unused fb_utilities
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 16:22:49 +02:00
Benny Halevy
f8a957898b tracing: use locator::topology rather than fb_utilities
Get my_address via query_processor->proxy and pass it
to all static make_ methods, instead of getting it from
utils::fb_utilities.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 16:22:15 +02:00
Benny Halevy
6f7de427f0 streaming: use locator::topology rather than fb_utilities
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 16:12:11 +02:00
Anna Stuchlik
409e20e5ab doc: enabling experimental Raft-managed topology
This commit adds a short paragraph to the Raft
page to explain how to enable consistent
topology updates with Raft - an experimental
feature in version 5.4.

The paragraph should satisfy the requirements
for version 5.4. The Raft page will be
rewritten in the next release when consistent
topology changes with Raft will be GA.

Fixes https://github.com/scylladb/scylladb/issues/15080

Requires backport to branch-5.4.

Closes scylladb/scylladb#16273
2023-12-05 14:49:17 +01:00
Yaniv Kaul
fef565482c Update unified/build_unified.sh
fix sentence overall
2023-12-05 15:23:38 +02:00
Yaniv Kaul
8f97429b16 Update main.cc
fix sentence overall, not just the typo
2023-12-05 15:21:48 +02:00
Yaniv Kaul
f2b810a16a Update dist/common/scripts/scylla-housekeeping
cobvert -> convert
2023-12-05 15:20:35 +02:00
Yaniv Kaul
ae2ab6000a Typos: fix typos in code
Fixes some more typos as found by codespell run on the code.
In this commit, there are more user-visible errors.

Refs: https://github.com/scylladb/scylladb/issues/16255
2023-12-05 15:18:11 +02:00
Botond Dénes
5fb0d667cb tools/scylla-sstable: always read scylla.yaml
Currently, scylla.yaml is read conditionally, if either the user
provided `--scylla-yaml-file` command line parameter, or if deducing the
data dir location from the sstable path failed.
We want the scylla.yaml file to be always read, so that when working
with encrypted file (enterprise), scylla-sstable can pick up the
configuration for the encryption.
This patch makes scylla-sstable always attempt to read the scylla-yaml
file, whether the user provided a location for it or not. When not, the
default location is used (also considering the `SCYLLA_CONF` and
`SCYLLA_HOME` environment variables.
Failing to find the scylla.yaml file is not considered an error. The
rational is that the user will discover this if they attempt to do an
operation that requires this anyway.
There is a debug-level log about whether it was successfully read or
not.

Fixes: #16132

Closes scylladb/scylladb#16174
2023-12-05 15:06:29 +02:00
Kefu Chai
2ebdc40b0b docs: add Deprecated to value_status_count
despite that the "value_status_count" is not rendered/used yet,
it'd be better to keep it in sync with the code.

since 5fd30578d7 added
"Deprecated" to `value_status` enum, let's update the sphinx
extension accordingly.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#16236
2023-12-05 14:52:13 +02:00
Avi Kivity
4498979b14 Merge 'When discarding table's sstables, delete them in one atomic batch' from Pavel Emelyanov
The table::discard_sstables() removes sstables attached to a table. For that it tries to atomically delete _each_ suitable sstable, which is a bit heavyweight -- each atomic deletion operation results in a deletion log file written. This PR deletes all table's sstables in one atomic batch. While at it, the body of the discard_sstables is simplified not to allocate the "pruner" object. The latter is possible after the method had become coroutine

Closes scylladb/scylladb#16202

* github.com:scylladb/scylladb:
  discard_sstables: Atomically delete all sstables
  discard_sstables: Indentation and formatting fix after previous patch
  discard_sstable: Open-code local prune() lambda
  discard_sstables: Do not allocate pruner
2023-12-05 14:17:06 +02:00
Benny Halevy
6c00c9a45d raft: use locator::topology/messaging rather than fb_utilities
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 13:26:46 +02:00
Benny Halevy
b3bede8141 storage_service: use locator::topology rather than fb_utilities
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 13:23:27 +02:00
Kamil Braun
52ae6b8738 Merge 'fix shutdown order between group0 and storage service' from Gleb
Storage service uses group0 internally, but group0 is create long after
storage service is initialized and passed to it using ss::set_group0()
function. What it means is that during shutdown group0 is destroyed
before ss::stop() is called and thus storage service is left with a
dangling reference. Fix it by introducing a function that cancels all
group0 operations and waits for background fibers to complete. For that
we need separate abort source for group0 operation which the patch
series also introduces.

* 'gleb/group0-ss-shutdown' of github.com:scylladb/scylla-dev:
  storage_service: topology coordinator: ignore abort_requested_exception in background fibers
  storage_service: fix de-initialization order between storage service and group0_service
2023-12-05 11:20:52 +01:00
Kefu Chai
e88bd9c5bd gms/inet_address: pass sstring param by std::move()
less overhead this way. the caller of lookup() always passes
a rvalue reference. and seastar::dns::get_host_by_name() actually
moves away from the parameter, so let's pass by std::move() for
slightly better performance, and to match the expectation of
the underlying seastar API.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#16280
2023-12-05 12:05:21 +03:00
Benny Halevy
a529097d96 storage_proxy: use locator::topology rather than fb_utilities
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 10:44:13 +02:00
Benny Halevy
0b310c471c service_level_controller: use locator::topology rather than fb_utilities
Expose cql3::query_processor in auth::service
to get to the topology via storage_proxy.replica::database

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 10:17:47 +02:00
Pavel Emelyanov
9bbbe7a99f discard_sstables: Atomically delete all sstables
When collected sstables are deleted each is passed into
sstables_manager.delete_atomically(). For on-disk sstables this creates
a deletion log for each removed stable, which is quite an overkill. The
atomic deletion callback already accepts vector of shared sstables, so
it's simpler (and a bit faster) to remove them all in a batch

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2023-12-05 11:14:23 +03:00
Pavel Emelyanov
96bc530a57 discard_sstables: Indentation and formatting fix after previous patch
By "formatting" fix I mean -- remove the temporary on-stack references
that were left for the ease of patching

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2023-12-05 11:13:40 +03:00
Pavel Emelyanov
6d135fea43 discard_sstable: Open-code local prune() lambda
The lambda in question was the struct pruner method and was left there
for the ease of patching. Now, when this lambda is only called once
inside the function it is declared in, it can be open-coded into the
place where it's called

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2023-12-05 11:13:40 +03:00
Pavel Emelyanov
68cb2e66fc discard_sstables: Do not allocate pruner
This allocation remained from the pre-coroutine times of the method. Now
the contents of prumer -- refernce on table, vector and replay_position
can reside on coroutine frame

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2023-12-05 11:13:40 +03:00
Benny Halevy
0e5754adc6 misc_services: use locator::topology rather than fb_utilities
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 10:01:36 +02:00
Benny Halevy
d49d10dbdb migration_manager: use messaging rather than fb_utilities
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 09:48:33 +02:00
Benny Halevy
860b2d38c6 forward_service: use messaging rather than fb_utilities
Use _forwarder._messaging to get to the broadcast address
rather than the global fb_utilities.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 09:48:12 +02:00
Benny Halevy
984a576405 messaging_service: accept broadcast_addr in config rather than via fb_utilities
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 09:46:25 +02:00
Benny Halevy
586f35bb55 messaging_service: move listen_address and port getters inline
And make them const noexcept.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 09:44:41 +02:00
Benny Halevy
eabd4570da test: manual: modernize message test
Basically, make it work (great) again.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 09:44:26 +02:00
Benny Halevy
f9acc90926 table: use gossiper rather than fb_utilities
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 09:43:47 +02:00
Benny Halevy
6826d87052 repair: use locator::topology rather than fb_utilities
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 09:09:06 +02:00
Benny Halevy
e1239e63bf dht/range_streamer: use locator::topology rather than fb_utilities
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 09:01:31 +02:00
Benny Halevy
63b556123b db/view: use locator::topology rather than fb_utilities
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 08:55:46 +02:00
Benny Halevy
f40bb7c583 database: use locator::topology rather than fb_utilities
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 08:42:49 +02:00
Benny Halevy
64145388c9 db/system_keyspace: use topology via db rather than fb_utilities
So not to rely on fb_utilities.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 08:42:49 +02:00
Benny Halevy
4bb4d673c3 db/system_keyspace: save_local_info: get broadcast addresses from caller
So not to rely on fb_utilities.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 08:42:49 +02:00
Benny Halevy
6e79d647e6 db/hints/manager: use locator::topology rather than fb_utilities
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 08:42:49 +02:00
Benny Halevy
4c20b84680 db/consistency_level: use locator::topology rather than fb_utilities
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 08:42:49 +02:00
Benny Halevy
e5d3c6741f api: use locator::topology rather than fb_utilities
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 08:42:49 +02:00
Benny Halevy
03fe674314 alternator: ttl: use locator::topology rather than fb_utilities
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 08:42:49 +02:00
Benny Halevy
f3e0358563 gossiper: use locator::topology rather than fb_utilities
And add `get_endpoint_state_ptr` for this_node.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 08:42:49 +02:00
Benny Halevy
25754f843b gossiper: add get_this_endpoint_state_ptr
Returns this node's endpoint_state_ptr.
With this entry point, the caller doesn't need to
get_broadcast_address.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 08:42:49 +02:00
Benny Halevy
21ace44f03 test: lib: cql_test_env: pass broadcast_address in cql_test_config
For getting rid of fb_utilities.

In the future, that could be used to instantiate
multiple scylla node instances.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 08:42:49 +02:00
Benny Halevy
3c846d3801 init: get_seeds_from_db_config: accept broadcast_address
Pass the broadcast_address from main to get_seeds_from_db_config
rather than getting it from fb_utilities.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 08:42:49 +02:00
Benny Halevy
4d461fc788 locator: replication strategies: use locator::topology rather than fb_utilities
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 08:42:49 +02:00
Benny Halevy
86716b2048 locator: topology: add helpers to retrieve this host_id and address
And respective `is_me()` predicates,
to prepare for getting rid of fb_utilities.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 08:42:49 +02:00
Benny Halevy
52412087b7 snitch: pass broadcast_address in snitch_config
To untangle snitch from fb_utilities.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 08:42:49 +02:00
Benny Halevy
94fc8e2a9a snitch: add optional get_broadcast_address method
and set broadcast_address / broadcast_rpc_address in main
to remove this dependency of snitch on fb_utilities.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 08:42:49 +02:00
Benny Halevy
1d0e71308b locator: ec2_multi_region_snitch: keep local public address as member
To be used in the next patch to retrieve the broadcast_address.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-12-05 08:42:49 +02:00