Commit Graph

172 Commits

Author SHA1 Message Date
Ran Regev
edd56a2c1c moved cache files to db
As requested in #22097, moved the files
and fixed other includes and build system.

Fixes: #22097
Signed-off-by: Ran Regev <ran.regev@scylladb.com>

Closes scylladb/scylladb#22495
2025-02-04 12:21:31 +03:00
Kefu Chai
5da2691f05 cmake: remove redundant BINARY_DIR setting for Seastar
ExternalProject automatically creates BINARY_DIR for Seastar, but generator
expressions are not supported in this setting. This caused CMake to create
an unused "build/$<CONFIG>/seastar" directory.

Instead, define a dedicated variable matching configure.py's naming and use
it in supported options like BUILD_COMMAND. This:

- Creates build files in the standard "Seastar-prefix/src/Seastar-build"
  directory instead of "build/$<CONFIG>/seastar". see
  https://cmake.org/cmake/help/latest/module/ExternalProject.html#directory-options
- Makes it clearer that the variable should match configure.py settings

No functional changes to the Seastar build process - purely a cleanup to
reduce confusion when inspecting the build directory.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#22437
2025-01-28 14:23:59 +03:00
Paweł Zakrzewski
384641194a audit: Add the audit subsystem
This change introduces a new audit subsystem that allows tracking and logging of database operations for security and compliance purposes. Key features include:

- Configurable audit logging to either syslog or a dedicated system table (audit.audit_log)
- Selective auditing based on:
  - Operation categories (QUERY, DML, DDL, DCL, AUTH, ADMIN)
  - Specific keyspaces
  - Specific tables
- New configuration options:
  - audit: Controls audit destination (none/syslog/table)
  - audit_categories: Comma-separated list of operation categories to audit
  - audit_tables: Specific tables to audit
  - audit_keyspaces: Specific keyspaces to audit
  - audit_unix_socket_path: Path for syslog socket
  - audit_syslog_write_buffer_size: Buffer size for syslog writes

The audit logs capture details including:
- Operation timestamp
- Node and client IP addresses
- Operation category and query
- Username
- Success/failure status
- Affected keyspace and table names
2025-01-15 11:10:35 +01:00
Avi Kivity
814942505f Merge 'Introduce Encryption-at-Rest (EAR) for sstables and commitlog' from Calle Wilund
Fixes https://github.com/scylladb/scylla-enterprise/issues/5016#issuecomment-2558464631

EAR - encryption at rest. Allows on-disk file encryption of sstables and commitlog data.
Introduces OpenSSL based file level encrypted storage, managed via a set of providers
ranging from local files to cloud KMS providers.

For a more comprehensive explanation, see the included docs (or if possible, original
source tree).

Manual bulk merge of EAR feature from enterprise repo to main scylla repo.

Breaks some features apart, but main EAR is still a humongous commit, because to separate this
I would have to mess with code incrementally, adding time and risk.

This PR includes the local file gen tool, tests and also p11 validation.

Note: CI will not execute the full tests unless master CI is set to provide the same environment
as the enterprise one. Not sure about the status of this ATM.

Note: Includes code to compile against cryptsoft kmipc SDK, but not the SDK. If you happen to
check out this tree in the scylla folder and configure, it will be linked against and KMIP functionality
will be enabled, otherwise not.

Closes scylladb/scylladb#22233

* github.com:scylladb/scylladb:
  docs: Add EAR docs
  main/build: Add p11-kit and initialize
  tools: Add local-file-key-generator tool
  tests: Add EAR tests
  tmpdir: shorten test tempdir path
  EAR: port the ear feature from enterprise
  cql_test_env: Add optional query timeout
  schema/migration_manager: Add schema validate
  sstables: add get_shared_components accessor
  config/config_file: Add exports and definitions of config_type_for<>
2025-01-12 16:10:46 +02:00
Piotr Smaron
288f9b2b15 Introduce LDAP role manager & saslauthd authenticator
This PR extends authentication with 2 mechanisms:
- a new role_manager subclass, which allows managing users via
LDAP server,
- a new authenticator, which delegates plaintext authentication
to a running saslauthd daemon.

The features have been ported from the enterprise repository
with their test.py tests and the documentation as part of
changing license to source available.

Fixes: scylladb/scylla-enterprise#5000
Fixes: scylladb/scylla-enterprise#5001

Closes scylladb/scylladb#22030
2025-01-12 14:50:29 +02:00
Calle Wilund
083f735366 main/build: Add p11-kit and initialize
For p11 certification/validation
2025-01-09 10:40:47 +00:00
Calle Wilund
723518c390 EAR: port the ear feature from enterprise
Bulk transfer of EAR functionality. Includes all providers etc.
Could maybe break up into smaller blocks, but once it gets down to
the core of it, would require messing with code instead of just moving.
So this is it.

Note: KMIP support is disabled unless you happen to have the kmipc
SDK in your scylla dir.

Adds optional encryption of sstables and commitlog, using block
level file encryption. Provides key sourcing from various sources,
such as local files or popular KMS systems.
2025-01-09 10:37:26 +00:00
Piotr Dulikowski
7383013f43 replica/database: add reader concurrency semaphore groups
Replace the reader concurrency semaphores for user reads and view
updates with the newly introduced reader concurrency semaphore group,
which assigns a semaphore for each service level.

Each group is statically assigned to some pool of memory on startup and
dynamically distribute this memory between the semaphores, relative to
the number of shares of the corresponding scheduling group.

The intent of having a separate reader concurrency semaphore for each
scheduling group is to prevent priority inversion issues due to reads
with different priorities waiting on the same semaphore, as well as make
memory allocation more fair between service levels due to the adjusted
number of shares.
2025-01-02 07:13:34 +01:00
Piotr Dulikowski
75d2d0d949 build: increase the max number of scheduling groups
Workload prioritization assigns scheduling groups to service levels, and
the number of scheduling groups that can exist at the same time is
limited with a compile-time parameter in seastar. The documentation for
workload prioritization says that we currently support 7 user-managed
service levels and 1 created by default. Increase the current
compile-time limit in order to align with the documentation.
2025-01-02 07:13:34 +01:00
Avi Kivity
76cf5148e1 Merge 'message: introduce advanced rpc compression' from Michał Chojnowski
This is a forward port (from scylla-enterprise) of additional compression options (zstd, dictionaries shared across messages) for inter-node network traffic. It works as follows:

After the patch, messaging_service (Scylla's interface for all inter-node communication)
compresses its network traffic with compressors managed by
the new advanced_rpc_compression::tracker. Those compressors compress with lz4,
but can also be configured to use zstd as long as a CPU usage limit isn't crossed.

A precomputed compression dictionary can be fed to the tracker. Each connection
handled by the tracker will then start a negotiation with the other end to switch
to this dictionary, and when it succeeds, the connection will start being compressed using that dictionary.

All traffic going through the tracker is passed as a single merged "stream" through dict_sampler.
dictionary_service has access to the dict_sampler.
On chosen nodes (in the "usual" configuration: the Raft leader), it uses the sampler to maintain
a random multi-megabyte sample of the sampler's stream. Every several minutes,
it copies the sample, trains a compression dictionary on it (by calling zstd's
training library via the alien_worker thread) and publishes the new dictionary
to system.dicts via Raft's write_mutation command.

This update triggers (eventually) a callback on all nodes, which feeds the new dictionary
to advanced_rpc_compression::tracker, and this switches (eventually) all inter-node connections
to this dictionary.

Closes scylladb/scylladb#22032

* github.com:scylladb/scylladb:
  messaging_service: use advanced_rpc_compression::tracker for compression
  message/dictionary_service: introduce dictionary_service
  service: make Raft group 0 aware of system.dicts
  db/system_keyspace: add system.dicts
  utils: add advanced_rpc_compressor
  utils: add dict_trainer
  utils: introduce reservoir_sampling
  utils: introduce alien_worker
  utils: add stream_compressor
2024-12-31 15:02:57 +02:00
Kefu Chai
6adf70ec03 build: cmake: add CMake options for PGO support
- "Scylla_BUILD_INSTRUMENTED" option

  Scylla_BUILD_INSTRUMENTED allows us to instrument the code at
  different level, namely, IR, and CSIR. this option mirrors
  "--pgo" and "--cspgo" options in `configure.py` . please note,
  the instrumentation at the frontend is not supported, as the IR
  based instrumentation is better when it comes to the use case of
  optimization for performance.
  see https://lists.llvm.org/pipermail/llvm-dev/2015-August/089044.html
  for the rationales.

- "Scylla_PROFDATA_FILE" option

  this option allows us to specify the profile data previous generated
  with the "Scylla_BUILD_INSTRUMENTED" option. this option mirrors
  the `--use-profile` option in `configure.py`, but it does not
  take the empty option as a special case and consider it as a file
  fetched from Git LFS. that will be handled by another option in a
  follow-up change. please note, one cannot use
  -DScylla_BUILD_INSTRUMENTED=PGO and -DScylla_PROFDATA_FILE=...
  at the same time. clang just does not allow this. but CSPGO is fine.

- "Scylla_PROFDATA_COMPRESSED_FILE" option

  this option allows us to specify the compressed profile data previouly
  generated with the "Scylla_BUILD_INSTRUMENTED" option. along with
  "Scylla_PROFDATA_FILE", this option mirros the functionality of
  `--use-profile` in `configure.py`. the goal is to ensure user always
  gets the result with the specified options. if anything goes wrong,
  we just error out.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-12-27 16:16:04 +08:00
Kefu Chai
4154789670 build: cmake: add "Scylla_ENABLE_LTO" option
add an option named "Scylla_ENABLE_LTO", which is off by default.
if it is on, build the whole tree with ThinLTO enabled.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-12-27 16:16:04 +08:00
Kefu Chai
ffe8c5dcdb build: collect scylla libraries with scylla_libs variable
with which, we can set the properties of these targets in a single place.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-12-27 16:16:04 +08:00
Kefu Chai
610f1b7a0a build: Unify Abseil CXX flags configuration
- Set ABSL_GCC_FLAGS and ABSL_LLVM_FLAGS with a more generic absl_cxx_flags
- Enables more flexible configuration of compiler flags for Abseil libraries
- Provides a centralized approach to setting compilation flags

Previously, sanitizer-specific flags were directly applied to Abseil library builds.
This change allows for more extensible compiling flag management across
different build configurations.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-12-27 16:16:04 +08:00
Michał Chojnowski
866326efe4 utils: add stream_compressor
Adds utilities for "advanced" methods of compression with lz4
and zstd -- with streaming (a history buffer persisted across messages)
and/or precomputed dictionaries.

This patch is mostly just glue needed to use the underlying
libraries with discontiguous input and output buffers, and for reusing the
same compressor context objects across messages. It doesn't contain
any innovations of its own.

There is one "design decision" in the patch. The block format of LZ4
doesn't contain the length of the compressed blocks. At decompression
time, that length must be delivered to the decompressor by a channel
separate to the compressed block itself. In `lz4_cstream`, we deal
with that by prepending a variable-length integer containing the
compressed size to each compressed block. This is suboptimal for
single-fragment messages, since the user of lz4_cstream is likely
going to remember the length of the whole message anyway,
which makes the length prepended to the block redundant.
But a loss of 1 byte is probably acceptable for most uses.
2024-12-23 23:28:12 +01:00
Botond Dénes
ca956c0180 configure: enable the io_uring backend
To be used by the tool apps -- also change the backend selected in
tools::utils::configure_tool_mode().
We keep using the more mature AIO backend in ScyllaDB itself, so main.cc
sets the linux_aio backend as the default one (the user can still change
this, same as before).
2024-12-04 02:55:31 -05:00
Kefu Chai
0f9d2ab577 build: cmake: disable Seastar exception hack
in cc3953e5, we disabled Seastar exception hack in configure.py.
this change disabled the Seastar exception hack in the following
two builds:

- build generated directly by configure.py
- build configured with multi-config generator using CMake

but we also have non-multi-config build using CMake. to be more
consistent, let's apply the equivalent change to non-multi-config
build of CMake.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#21233
2024-10-28 08:11:43 +02:00
Kefu Chai
f5b29331a2 build: populate --enable-dist --disable-dist to CMake
before this change, the "dist" targets are always enabled in the
CMake-based building system. but the build rules generated by
`configure.py` does respect `--enable-dist` and `--disable-dist`
command line options, and enable/distable the dist targets
respectively.

in this change, we

- add an CMake option named "Scylla_DIST". the "dist"
  subdirectory in CMake only if this option is ON.
- pouplate the `--enable-dist` and `--disable-dist` option
  down to cmake by setting the `Scylla_DIST` option,
  when creating the build system using CMake.

this enables the CMake-based build system to be functionality
wise more closer to the legacy building system.

Refs scylladb/scylladb#2717

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#21253
2024-10-27 21:57:46 +02:00
Kefu Chai
e73b0c942f build: cmake: use GENERATOR_IS_MULTI_CONFIG property to detect mult-config
this is more reliable way to check if we are configured to use a
mult-config generator.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-10-18 08:36:52 +08:00
Kefu Chai
415c83fa67 build: cmake: consume Seastar using its .pc files
before this change, scylla's CMake-based system consumes Seastar
library by including it directly. but this failed to address the needs
of linking against Seastar shared libraries in Debug and Dev builds, while
linking against the static libraries in other builds. because Seastar
uses `BUILD_SHARED_LIBS` CMake variable to determine if it builds
shared libraries. and we cannot assign different values to this
CMake variable based on current configure type -- CMake does not
support. see https://gitlab.kitware.com/cmake/cmake/-/issues/19467

in order to address this problem, we have a couple possible
solutions:

- to enable Seastar to build both shared and static libraries in a
  pass. without sacrificing the performance, we have to build
  all object files twice: once with -fPIC, once without. in order
  to accompolish this goal, we need to develop a machinary to
  populate the same settings to these two builds. this would
  complicate the design of Seastar's building system further.
- to build Seastar libraries twice in scylla, we could use
  the ExternalProject module to implement this. but it'd be
  complicate to extract the compile options, and link options
  previously populated by Seastar's targets with CMake --
  we would have to replicate all of them in scylla. this is
  out of the question.
- to build Seastar libraries twice before building scylla,
  and let scylla to consume them using CMake config files or
  .pc files. this is a compromise. it enables scylla to
  drive the build of Seastar libraries and to consume
  the compile options and link options. the downside is:

  * the generated compilation database (compile_commands.json)
    does not include the commands building Seastar anymore.
  * the building system of scylla does not have finer graind
    control on the building process of seastar. for instance,
    we cannot specify the build dependency to a certain seastar
    library, and just build it instead of building the whole
    seastar project.

turns out the last approach is the best one we can have
at this moment. this is also the approach used by the existing
`configure.py`.

in this change, we

- add FindSeastar.cmake to

  * detect the preconfigured Seastar builds, and
  * extract the build options from .pc files
  * expose library targets to be consumed by parent project
- add Seastar as an external project, so we can build it from
  the parent project. BUILD_AWAYS is set to ensure that Seastar is
  rebuilt, as scylla developers are expected to modify Seastar
  occasionally. since the change in Seastar's SOURCE_DIR is not
  detectable via the ExternalProject, we have to rebuild it.

  this is atypical compared to standard ExternalProject usage:
  - Seastar's build system should already be configured at this point.
  - We maintain separate project variants for each configuration type.

  Benefits of this approach:
  - Allows the parent project to consume the compile options exposed by
    .pc file. as the compile options vary from one config to another.
  - Allows application of config-specific settings
  - Enables building Seastar within the parent project's build system
  - Facilitates linking of artifacts with the external project target,
    establishing proper dependencies between them
- preserve the existing machinery of including Seastar only when
  building without multi-config generator. this allows users who don't
  use mult-config generator to build Seastar in-the-tree. the typical
  use case is the CI workflows performing the static analysis.

we will update `configure.py` to merge the compilation database
of scylla and seastar.

Refs scylladb/scylladb#2717
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-10-18 08:36:52 +08:00
Kefu Chai
fc8212483e build: cmake: detect and link against yaml-cpp
in main.cc, we use yaml-cpp library directly. so we are obliged to
detect this library in scylla and link against it instead of relying
on other library to do this. currently, Seastar detects it and pulls
in yaml-cpp for us, but we should not take this for granted and rely
on this.

in this change, we detect and link against yaml-cpp to make this
dependency explicit.

the same applies to the "utils" library.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-10-18 08:36:52 +08:00
Kefu Chai
2e4be56112 build: cmake: link Seastar with Seastar::<COMPONENT>
before this change, we link against the targets defined in Seastar's
source tree. but these targets are not part of Seastar's public
interface -- they are not exposed by Seastar's CMake config files.

so, let link against the target names qualified by the library module
name. this also prepares for the transition to using Seastar without
including it directly.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-10-18 08:36:52 +08:00
Kefu Chai
e4b213f041 build: cmake: use the same options to configure seastar
in `configure.py`, a set of options are specified when configuring
seastar, but not all of them were ported to scylla's CMake building
system.

for instance, `configure.py` explicitly disables io_uring reactor
backend at build time, but the CMake-based system does not.

so, in this change, in order to preserve the existing behavior, let's
port the two previously missing option to CMake-based building system
as well.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#20288
2024-08-28 06:15:59 +03:00
Kefu Chai
8ef26a9c8c build: cmake: add "test" target
before this change, none of the target generated by CMake-based
building system runs `test.py`. but `build.ninja` generated directly
by `configure.py` provides a target named `test`, which runs the
`test.py` with the options passed to `configure.py`.

to be more compatible with the rules generated by `configure.py`,
in this change

* do not include "CTest" module, as we are not using CTest for
  driving tests. we use the homebrew `test.py` for this purpose.
  more importantly, the target named "test" is provided by "CTest".
  so in order to add our own "test" target, we cannot use "CTest"
  module.
* add a target named "test" to run "test.py".
* add two CMake options so we can customize the behavior of "test.py",
  this is to be compatible with the existing behavior of `configure.py`.

Refs scylladb/scylladb#2717

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#20263
2024-08-25 21:45:13 +03:00
Kefu Chai
5ce07e5d84 build: cmake: add compiler-training target
`tools/toolchain/optimized_clang.sh` builds this target for creating
the profile in order to build clang optimized with this profile data.

so let's be compatible with `configure.py`, and add this target to
CMake building system as well.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#20105
2024-08-14 21:21:33 +03:00
Kefu Chai
0ae04ee819 build: cmake: use $<CONFIG:cfgs> when appropriate
per
https://cmake.org/cmake/help/latest/manual/cmake-generator-expressions.7.html#genex:CONFIG,
`cfgs` can be a comma-separated list. this is supported by CMake
3.19 and up, and our minimum required CMake version is 3.27. so let's
switch over from the composition of `IN_LIST` and `CONFIG` generator
expressions to a single one. simpler this way.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#20110
2024-08-11 21:28:38 +03:00
Aleksandra Martyniuk
91fbfbf98a node_ops: add task manager module and node_ops_virtual_task
Add task manager node ops module and node_ops_virtual_task.

Some methods will be implemented in later patches.
2024-07-23 13:35:01 +02:00
Aleksandra Martyniuk
1d85b319e0 tasks: add task_handler to hide task and virtual_task differences from user
Contrary to regular tasks, which are per-operation, virtual tasks
are associated with the whole group of operations. There may be many
operations of each group performed at the same time. Info about each
running operation will be shown to a user through the API.

For virtual tasks, task manager imitates a regular task covering
each operation, but task_manager::tasks aren't actually created
in the memory. Instead, information (e.g. status) about the operation
is retrieved from associated service and passed to a user.

To hide most of the differences from user, task_handler class is created.
Task handler performs appropriate actions depending on task's kind.

However, users need to stay conscious about the kind of task, because:
- get_task_status and wait_task do not unregister virtual tasks;
- time for which a virtual tasks stays in task manager depends
  on associated service and tasks' implementation;
- number of virtual task's children shown by get_tasks doesn't have
  to be monotonous.

API is modified to use task_handler.
API-specific classes are moved to task_handler.{cc,hh}.
2024-07-23 13:35:01 +02:00
Kefu Chai
34bf10050b build: cmake: bump up the minimal required fmt to 10.0.0
in cccec07581, we started using a featured introduced by {fmt} v10.
so we need to bump up the required version in CMake as well.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#19709
2024-07-19 07:58:31 +03:00
Kefu Chai
90be71d959 build: cmake: include the whole archive of zstd.a
before this change, when linking scylla-main, the linker discards
the unreferenced symbols defined by zstd.cc. but we use constructor
of static variable `registerator` to register the zstd compressor,
this variable is not used from the linker's point of view. but we
do rely on the side effect of its constructor.
that's why the rules generated by CMake fails to build tests and
scylla executables with zstd support. that's why we have following
test failure:
```
boost.sstable_3_x_test.test_uncompressed_collections_read
...
[Exception] - no_such_class: unable to find class 'org.apache.cassandra.io.compress.ZstdCompressor'
 == [File] - seastar/src/testing/seastar_test.cc
 == [Line] - 43
```

in this change, we single out zstd.cc and build it as an archive,
so that scylla-main can include as a whole. an alternative is to
link scylla-main as a whole archive, but that might increase the disk
foot print when building lots of tests -- some of them do not use all
symbols exposed by scylla-main, and can potentially have smaller
size if linker can discard the unused symbols.

Refs #2717
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-07-01 11:51:19 +08:00
Kefu Chai
1e0af0fb7e build: cmake: find libzstd before using it
we use libzstd in zstd.cc. so let's find this library before using
it. this helps user to identify problem when preparing the building
environment, instead of being greeted by a compile-time failure.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-07-01 11:51:19 +08:00
Avi Kivity
4d85db9f39 build: switch to C++23
Set the C++ dialect to C++23, allowing us to use the new features.
2024-06-27 19:36:13 +03:00
Kefu Chai
ad649be1bf treewide: drop thrift support
thrift support was deprecated since ScyllaDB 5.2

> Thrift API - legacy ScyllaDB (and Apache Cassandra) API is
> deprecated and will be removed in followup release. Thrift has
> been disabled by default.

so let's drop it. in this change,

* thrift protocol support is dropped
* all references to thrift support in document are dropped
* the "thrift_version" column in system.local table is
  preserved for backward compatibility, as we could load
  from an existing system.local table which still contains
  this clolumn, so we need to write this column as well.
* "/storage_service/rpc_server" is only preserved for
  backward compatibility with java-based nodetool.
* `rpc_port` and `start_rpc` options are preserved, but
  they are marked as "Unused". so that the new release
  of scylladb can consume existing scylla.yaml configurations
  which might contain these settings. by making them
  deprecated, user will be able get warned, and update
  their configurations before we actually remove them
  in the next major release.

Fixes #3811
Fixes #18416
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-06-07 06:44:59 +08:00
Kefu Chai
0da0461668 build: cmake: do not scan for C++20 modules
when creating the build rules using CMake 3.28 and up, it generates
the rules to scan for C++20 modules for C++20 projects by default.
but this slows down the compilation, and introduces unnecessary
dependencies for each of the targets when building .cc files. also,
it prevents the static analysis tools from running from a repo which
only have its building system generated, but not yet built. as,
these tools would need to process the source files just like a compiler
does, and if any of the included header files is missing, they just
fail.

so, before we migrate to C++20 modules, let's disable this feature.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#19038
2024-06-03 12:51:40 +03:00
Kefu Chai
9bd9f283f4 build: disable operator<< for vector and unordered_map
seastar provides an option named `Seastar_DEPRECATED_OSTREAM_FORMATTERS`
to enable the operator<< for `std::vector` and `std::unordered_map`,
and this option is enabled by default. but we intent to avoid using
them, so that we can use the fmt::formatter specializations when
Boost.test prints variables. if we keep these two operator<< enabled,
Boost.test would use them when printing variables to be compaired
then the check fails, but if elements in the vector or unordered_map
to be compaired does do not provide operator<<, compiling would fail.

so, in this change, let's disable these operator<< implementations.
this allows us to ditch the operator<< implementations which are
preserved only for testing.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-05-26 13:44:55 +08:00
Kefu Chai
c336904722 build: cmake: mark abseil include SYSTEM
this change is a followup of 0b0e661a. it helps to ensure that the header files in
abseil submodule have higher priority when the compiler includes abseil headers
when building with CMake.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#18523
2024-05-09 08:26:44 +03:00
Kefu Chai
0b0e661a85 build: bring abseil submodule back
because of https://bugzilla.redhat.com/show_bug.cgi?id=2278689,
the rebuilt abseil package provided by fedora has different settings
than the ones if the tree is built with the sanitizer enabled. this
inconsistency leads to a crash.

to address this problem, we have to reinstate the abseil submodule, so
we can built it with the same compiler options with which we build the
tree.

in this change

* Revert "build: drop abseil submodule, replace with distribution abseil"
* update CMake building system with abseil header include settings
* bump up the abseil submodule to the latest LTS branch of abseil:
  lts_2024_01_16
* update scylla-gdb.py to adapt to the new structure of
  flat_hash_map

This reverts commit 8635d24424.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#18511
2024-05-05 23:31:09 +03:00
Kefu Chai
014a069ed2 build: cmake: require {fmt} >= 9.0.0
we are using `fmt::ostream_formatter` which was introduced in
{fmt} v9.0.0, see https://github.com/fmtlib/fmt/releases/tag/9.0.0 .

before this change, we depend on Seastar to find {fmt}. but
the minimal version of {fmt} required by Seastar is 5.0.0, which
cannot fulfill the needs to build scylladb.

in this change, we find {fmt} package in scylla, and specify the
minimal required version of 9.0.0, so the build can fail at the
configuration time. {fmt} v8 could be still being used by users.
for instance, ubuntu:jammy comes with libfmt-dev 8.1.1. and
ubuntu:jammy is EOL in Apr 2027, see
https://ubuntu.com/about/release-cycle .

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#18386
2024-04-25 16:35:08 +03:00
Kefu Chai
372a4d1b79 treewide: do not define FMT_DEPRECATED_OSTREAM
since we do not rely on FMT_DEPRECATED_OSTREAM to define the
fmt::formatter for us anymore, let's stop defining `FMT_DEPRECATED_OSTREAM`.

in this change,

* utils: drop the range formatters in to_string.hh and to_string.c, as
  we don't use them anymore. and the tests for them in
  test/boost/string_format_test.cc are removed accordingly.
* utils: use fmt to print chunk_vector and small_vector. as
  we are not able to print the elements using operator<< anymore
  after switching to {fmt} formatters.
* test/boost: specialize fmt::details::is_std_string_like<bytes>
  due to a bug in {fmt} v9, {fmt} fails to format a range whose
  element type is `basic_sstring<uint8_t>`, as it considers it
  as a string-like type, but `basic_sstring<uint8_t>`'s char type
  is signed char, not char. this issue does not exist in {fmt} v10,
  so, in this change, we add a workaround to explicitly specialize
  the type trait to assure that {fmt} format this type using its
  `fmt::formatter` specialization instead of trying to format it
  as a string. also, {fmt}'s generic ranges formatter calls the
  pair formatter's `set_brackets()` and `set_separator()` methods
  when printing the range, but operator<< based formatter does not
  provide these method, we have to include this change in the change
  switching to {fmt}, otherwise the change specializing
  `fmt::details::is_std_string_like<bytes>` won't compile.
* test/boost: in tests, we use `BOOST_REQUIRE_EQUAL()` and its friends
  for comparing values. but without the operator<< based formatters,
  Boost.Test would not be able to print them. after removing
  the homebrew formatters, we need to use the generic
  `boost_test_print_type()` helper to do this job. so we are
  including `test_utils.hh` in tests so that we can print
  the formattable types.
* treewide: add "#include "utils/to_string.hh" where
  `fmt::formatter<optional<>>` is used.
* configure.py: do not define FMT_DEPRECATED_OSTREAM
* cmake: do not define FMT_DEPRECATED_OSTREAM

Refs #13245

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-04-19 22:57:36 +08:00
Kefu Chai
a0625261ef build: cmake: reword the comment for dev-headers
before this change, the comment was difficult to parse. let's update
it for better readability.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#17814
2024-03-15 09:51:47 +02:00
Avi Kivity
3ab2088119 Merge 'build: cmake: use scylla build mode for rust profile name ' from Kefu Chai
before this change, we used the lower-case CMake build configuration
name for the rust profile names. but this was wrong, because the
profiles are named with the scylla build mode.

in this change, we translate the `$<CONFIG>` to scylla build mode,
and use it for the profile name and for the output directory of
the built library.

Closes scylladb/scylladb#17648

* github.com:scylladb/scylladb:
  build: cmake: use scylla build mode for rust profile name
  build: cmake: define per-config build mode
2024-03-06 13:46:20 +02:00
Kefu Chai
4d4c0ddf31 build: cmake: exclude Seastar's tests from "all"
in 02de9f1833, we enable building Seastar testing for using the
testing facilities in scylla's own tests. but this brings in
Seastar's tests.

since scylladb's CI builds the "all" targets, and we are not
interested in running Seastar's tests when building scylladb,
let's exclude Seastar's tests from the "all" target.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#17554
2024-03-06 10:15:45 +02:00
Kefu Chai
0c1864eebd build: cmake: define per-config build mode
so that scylla_build_mode_$<CONFIG> can be referenced when necessary.
we using it for referencing build mode in the building system instead
of the CMake configuration name.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2024-03-06 15:53:11 +08:00
Kefu Chai
2bec6751d3 build: cmake: add "mode_list" target
scylla uses build modes like "debug" and "release" to differentiate
different build modes. while we intend to use the typical build
configurations / build types used by CMake like "Debug" and
"RelWithDebInfo" for naming CMAKE_CONFIGURATION_TYPES and
CMAKE_BUILD_TYPE. the former is used for naming the build directory and
for the preprocess macro named "SCYLLA_BUILD_MODE".

`test.py` and scylladb's CI are designed based on the naming of build
directory. in which, `test.py` lists the build modes using the dedicated
build target named `list_modes`, which is added by `configure.py`.

so, in this change, the target is added to CMake as well. the variables
of "scylla_build_mode" defined by the per-mode configuration are
collected and printed by the `list_modes`.

because, by default, CMake generates a target for each build
configuration when a multi-config generator is used. but we only want to
print the build mode for a single time when "list_modes" is built. so
a "BYPRODUCTS" is deliberately added for the target, and the patch of
this "BYPRODUCTS" is named without the "$<CONFIG>" it its path.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2023-12-24 12:35:02 +08:00
Kefu Chai
fafe9d9c38 build: cmake: map 'release' to 'RelWithDebInfo'
this preserves the existing behavior of `configure.py` in the CMake
generated `build.ninja`.

* configure.py: map 'release' to 'RelWithDebInfo'
* cmake: rename cmake/mode.Release.cmake to cmake/mode.RelWithDebInfo.cmake
* CMakeLists.txt: s/Release/RelWithDebInfo/

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2023-12-20 15:07:43 +08:00
Kefu Chai
65994b1e83 build: cmake: add compatibility target of dev-headers
our CI builds "dev-headers" as a gating check. but the target names
generated by CMake's Ninja Multi-Config generator does not follow
this naming convention. we could have headers:Dev, but still, it's
different from what we are using, before completely switching to
CMake, let's keep this backward compatibility by adding a target
with the same name.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2023-11-29 10:08:59 +08:00
Kefu Chai
efd65aebb2 build: cmake: add check-header target
to have feature parity with `configure.py`. we won't need this
once we migrate to C++20 modules. but before that day comes, we
need to stick with C++ headers.

we generate a rule for each .hh files to create a corresponding
.cc and then compile it, in order to verify the self-containness of
that header. so the number of rule is quite large, to avoid the
unnecessary overhead. the check-header target is enabled only if
`Scylla_CHECK_HEADERS` option is enabled.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#15913
2023-11-13 10:27:06 +02:00
Kefu Chai
61a542ffd0 build: cmake: configure all available config types
if `CMAKE_CONFIGURATION_TYPES` is set, it implies that the
multi-config generator is used, in this case, we include all
available build types instead of only the one specified by
`CMAKE_BUILD_TYPE`, which is typically used by non-multi-config
generators.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2023-11-07 10:14:33 +08:00
Kefu Chai
6fcff51cf1 build: cmake: set per-mode stack usage threshold
instead of setting a single stack usage threshold, set per-mode
stack usage threshold. this prepares for the support of
multi-config generator.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2023-11-07 10:13:50 +08:00
Kefu Chai
23bb644314 build: cmake: drop build_mode
there is no benefit having this variable. and it introduces
another layer of indirection. so drop it.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2023-11-07 10:10:59 +08:00