Move dev docs to docs/dev

HACKING.md

@@ -419,4 +419,4 @@ $ eu-unstrip -n --core <coredump> | awk '/scylla$/ { s=$2; sub(/@.*$/, "", s); p
 
 ### Core dump debugging
 
-See [debugging.md](docs/guides/debugging.md).
+See [debugging.md](docs/dev/debugging.md).

README.md

@@ -42,7 +42,7 @@ For further information, please see:
 * [Docker image build documentation] for information on how to build Docker images.
 
 [developer documentation]: HACKING.md
-[build documentation]: docs/guides/building.md
+[build documentation]: docs/dev/building.md
 [docker image build documentation]: dist/docker/debian/README.md
 
 ## Running Scylla
@@ -65,7 +65,7 @@ $ ./tools/toolchain/dbuild ./build/release/scylla --help
 
 ## Testing
 
-See [test.py manual](docs/guides/testing.md).
+See [test.py manual](docs/dev/testing.md).
 
 ## Scylla APIs and compatibility
 By default, Scylla is compatible with Apache Cassandra and its APIs - CQL and
@@ -73,12 +73,12 @@ Thrift. There is also support for the API of Amazon DynamoDB™,
 which needs to be enabled and configured in order to be used. For more
 information on how to enable the DynamoDB™ API in Scylla,
 and the current compatibility of this feature as well as Scylla-specific extensions, see
-[Alternator](docs/alternator/alternator.md) and
-[Getting started with Alternator](docs/alternator/getting-started.md).
+[Alternator](docs/source/alternator/alternator.md) and
+[Getting started with Alternator](docs/source/alternator/getting-started.md).
 
 ## Documentation
 
-Documentation can be found [here](https://scylla.docs.scylladb.com).
+Documentation can be found [here](docs/dev/README.md).
 Seastar documentation can be found [here](http://docs.seastar.io/master/index.html).
 User documentation can be found [here](https://docs.scylladb.com/).
 

docs/Makefile

@@ -1,19 +1,17 @@
+# Global variables
 # You can set these variables from the command line.
-POETRY        := poetry
-SPHINXBUILD   := $(POETRY) run sphinx-build
-SPHINXOPTS    :=
-PAPER         :=
-BUILDDIR      := _build
-SOURCEDIR     := .
+POETRY        = poetry
+SPHINXOPTS    =
+SPHINXBUILD   = $(POETRY) run sphinx-build
+PAPER         =
+BUILDDIR      = _build
+SOURCEDIR     = source
 
-# Internal variables.
-PAPEROPT_a4     := -D latex_paper_size=a4
-PAPEROPT_letter := -D latex_paper_size=letter
-ALLSPHINXOPTS   := -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) $(SOURCEDIR)
-TESTSPHINXOPTS  := $(ALLSPHINXOPTS) -W --keep-going
-
-# the i18n builder cannot share the environment and doctrees with the others
-I18NSPHINXOPTS  := $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) $(SOURCEDIR)
+# Internal variables
+PAPEROPT_a4     = -D latex_paper_size=a4
+PAPEROPT_letter = -D latex_paper_size=letter
+ALLSPHINXOPTS   = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) $(SOURCEDIR)
+TESTSPHINXOPTS  = $(ALLSPHINXOPTS) -W --keep-going
 
 # Windows variables
 ifeq ($(OS),Windows_NT)
@@ -69,7 +67,7 @@ epub3: setup
 
 .PHONY: multiversion
 multiversion: setup
-	$(POETRY) run sphinx-multiversion $(SOURCEDIR) $(BUILDDIR)/dirhtml
+	$(POETRY) run sphinx-multiversion source $(BUILDDIR)/dirhtml
 	@echo
 	@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
 
@@ -91,4 +89,4 @@ test: setup
 
 .PHONY: linkcheck
 linkcheck: setup
-	$(SPHINXBUILD) -b linkcheck $(SOURCEDIR) $(BUILDDIR)/linkcheck
+	$(SPHINXBUILD) -b linkcheck $(SOURCEDIR) $(BUILDDIR)/linkcheck

docs/README.md

@@ -1,15 +0,0 @@
-# Scylla Developer Documentation
-
-This documentation targets developers who are interested in contributing to Scylla codebase.
-
-## Contents
-
-* [Alternator](alternator/alternator.md) - The open source DynamoDB-compatible API reference.
-* [Design notes](design-notes/index.md) - Explanations describing how new features work for other contributors and exploratory research.
-* [Guides](guides/index.md) - Instructions on how to build, run, test and debug the Scylla codebase.
-* [Contribute](contribute/index.md) - Guidelines on how to contribute and maintain the project.
-
-## Other Useful Documents and Links
-
-* [User documentation](https://docs.scylladb.com/)
-* [Seastar documentation](http://docs.seastar.io/master/index.html)

docs/conf.py

@@ -1,150 +0,0 @@
-# -*- coding: utf-8 -*-
-import os
-import sys
-from datetime import date
-import recommonmark
-from recommonmark.transform import AutoStructify
-from pygments.lexers.javascript import JavascriptLexer
-from sphinx_scylladb_theme.utils import multiversion_regex_builder
-
-sys.path.insert(0, os.path.abspath('..'))
-
-# -- General configuration ------------------------------------------------
-
-# Build documentation for the following tags and branches
-TAGS = []
-BRANCHES = ['branch-4.5', 'branch-4.6', 'master']
-# Set the latest version.
-LATEST_VERSION = 'branch-4.6'
-# Set which versions are not released yet.
-UNSTABLE_VERSIONS = ['master']
-# Set which versions are deprecated
-DEPRECATED_VERSIONS = ['']
-
-# Add any Sphinx extension module names here, as strings. They can be
-# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
-# ones.
-extensions = [
-    'sphinx.ext.todo',
-    'sphinx.ext.mathjax',
-    'sphinx.ext.githubpages',
-    'sphinx.ext.extlinks',
-    'sphinx_sitemap',
-    'sphinx_scylladb_theme',
-    'sphinx_multiversion',
-    'recommonmark',
-    'sphinx_markdown_tables',
-]
-
-# The suffix(es) of source filenames.
-# You can specify multiple suffix as a list of string:
-#
-source_suffix = ['.rst', '.md']
-autosectionlabel_prefix_document = True
-
-# The master toctree document.
-master_doc = 'contents'
-
-# General information about the project.
-project = 'Scylla Dev'
-copyright = str(date.today().year) + ', ScyllaDB. All rights reserved.'
-author = u'Scylla Project Contributors'
-
-# List of patterns, relative to source directory, that match files and
-# directories to ignore when looking for source files.
-# This patterns also effect to html_static_path and html_extra_path
-exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store', 'README.md', '_utils']
-
-# The name of the Pygments (syntax highlighting) style to use.
-pygments_style = 'sphinx'
-
-# -- Options for not found extension -------------------------------------------
-
-# Template used to render the 404.html generated by this extension.
-notfound_template =  '404.html'
-
-# Prefix added to all the URLs generated in the 404 page.
-notfound_urls_prefix = ''
-
-# -- Options for redirect extension ---------------------------------------
-
-# Read a YAML dictionary of redirections and generate an HTML file for each
-redirects_file = '_utils/redirections.yaml'
-
-# -- Options for multiversion extension ----------------------------------
-# Whitelist pattern for tags
-smv_tag_whitelist = multiversion_regex_builder(TAGS)
-# Whitelist pattern for branches
-smv_branch_whitelist = multiversion_regex_builder(BRANCHES)
-# Defines which version is considered to be the latest stable version.
-smv_latest_version = LATEST_VERSION
-# Defines the new name for the latest version.
-smv_rename_latest_version = 'stable'
-# Whitelist pattern for remotes (set to None to use local branches only)
-smv_remote_whitelist = r'^origin$'
-# Pattern for released versions
-smv_released_pattern = r'^tags/.*$'
-# Format for versioned output directories inside the build directory
-smv_outputdir_format = '{ref.name}'
-
-# -- Options for sitemap extension ---------------------------------------
-
-sitemap_url_scheme = 'stable/{link}'
-
-# -- Options for HTML output ----------------------------------------------
-
-# The theme to use for HTML and HTML Help pages.  See the documentation for
-# a list of builtin themes.
-#
-html_theme = 'sphinx_scylladb_theme'
-
-# Theme options are theme-specific and customize the look and feel of a theme
-# further.  For a list of options available for each theme, see the
-# documentation.
-#
-html_theme_options = {
-    'conf_py_path': 'docs/',
-    'default_branch': 'master',
-    'github_repository': 'scylladb/scylla',
-    'github_issues_repository': 'scylladb/scylla',
-    'hide_edit_this_page_button': 'false',
-    'hide_version_dropdown': ['master'],
-    'versions_unstable': UNSTABLE_VERSIONS,
-    'versions_deprecated': DEPRECATED_VERSIONS,
-}
-
-# If not None, a 'Last updated on:' timestamp is inserted at every page
-# bottom, using the given strftime format.
-# The empty string is equivalent to '%b %d, %Y'.
-#
-html_last_updated_fmt = '%d %B %Y'
-
-# Custom sidebar templates, maps document names to template names.
-#
-html_sidebars = {'**': ['side-nav.html']}
-
-# Output file base name for HTML help builder.
-htmlhelp_basename = 'ScyllaDocumentationdoc'
-
-# URL which points to the root of the HTML documentation. 
-html_baseurl = 'https://scylla.docs.scylladb.com'
-
-# Dictionary of values to pass into the template engine’s context for all pages
-html_context = {'html_baseurl': html_baseurl}
-
-
-class AssemblyScriptLexer(JavascriptLexer):
-    pass
-
-# Setup Sphinx
-def setup(sphinx):
-    # Add Markdown support
-    sphinx.add_config_value('recommonmark_config', {
-        'enable_eval_rst': True,
-        'enable_auto_toc_tree': False,
-    }, True)
-    sphinx.add_transform(AutoStructify)
-    
-    # Custom lexers
-    sphinx.add_lexer('assemblyscript', AssemblyScriptLexer)
-

docs/contents.rst

@@ -1,9 +0,0 @@
-.. toctree::
-    :hidden:
-    :maxdepth: 2
-    
-    index
-    alternator/alternator
-    design-notes/index
-    guides/index
-    contribute/index

docs/contribute/CONTRIBUTING.md

@@ -1 +0,0 @@
-../../CONTRIBUTING.md

docs/contribute/index.md

@@ -1,13 +0,0 @@
-# Contribute 
-
-Guidelines on how to contribute and maintain the project.
-
-## Contents
-
-```eval_rst
-.. toctree::
-   :glob:
-   :maxdepth: 1
-
-   *
-```

docs/design-notes/index.md

@@ -1,13 +0,0 @@
-# Design Notes
-
-Explanations describing how new features work for other contributors and exploratory research.
-
-## Contents
-
-```eval_rst
-.. toctree::
-   :glob:
-   :maxdepth: 1
-   
-   *
-```

docs/dev/README.md

@@ -0,0 +1,16 @@
+# Scylla developer documentation
+
+This folder (and its subfolders) contain developer-oriented documentation
+concerning the Scylla codebase.
+We also have a [wiki](https://github.com/scylladb/scylla/wiki), which contains
+additional developer-oriented documentation. There is currently no clear
+definition of what goes where, so when looking for something be sure to check
+both.
+
+Seastar documentation can be found [here](http://docs.seastar.io/master/index.html).
+
+User documentation can be found on
+[docs.scylladb.com](https://docs.scylladb.com/)
+
+For information on how to build Scylla and how to contribute visit
+[HACKING.md](../../HACKING.md) and [CONTRIBUTING.md](../../CONTRIBUTING.md).

docs/dev/docker-hub.md

@@ -206,7 +206,7 @@ Note that the `--alternator-https-port` option also requires that files `/etc/sc
 
 #### `--alternator-write-isolation policy`
 
-The `--alternator-write-isolation` command line option chooses between four allowed write isolation policies described in docs/alternator/alternator.md. This option must be specified if Alternator is enabled - it does not have a default.
+The `--alternator-write-isolation` command line option chooses between four allowed write isolation policies described in docs/source/alternator/alternator.md. This option must be specified if Alternator is enabled - it does not have a default.
 
 **Since: 4.1**
 

docs/dev/maintainer.md

@@ -268,7 +268,7 @@ still has to rely on their judgemenet.
 2. Have a subject matter expert review the code. If you are
    not familiar with the modified code, ask one to do a
    review. For more details on how to review patches, see
-   the [review checklist](https://github.com/scylladb/scylla/blob/master/docs/contribute/review-checklist.md).
+   the [review checklist](review-checklist.md).
 
 3. If the pull request fixes an issue, it should have a
    Fixes or Refs depending on whether the fix is complete or

docs/dev/protocols.md

@@ -197,7 +197,7 @@ HTTP (unencrypted) or HTTPS (encrypted) protocol. Support for this protocol
 is not turned on by default, and must be turned on manually by setting
 the `alternator_port` and/or `alternator_https_port` configuration options.
 "Alternator" is the codename of Scylla's DynamoDB API support, and is
-documented in more detail in [alternator.md](../alternator/alternator.md).
+documented in more detail in [alternator.md](../source/alternator/alternator.md).
 
 The standard ports that DynamoDB uses are the standard HTTP and HTTPS
 ports (80 and 443, respectively), but in tests we usually use the

docs/dev/system_keyspace.md

@@ -153,7 +153,7 @@ legacy records. When the feature is agreed upon the legacy map is removed.
 Virtual tables behave just like a regular table from the user's point of view.
 The difference between them and regular tables comes down to how they are implemented.
 While regular tables have memtables/commitlog/sstables and all you would expect from CQL tables, virtual tables translate some in-memory structure to CQL result format.
-For more details see the [docs/guides/virtual-tables.md](../guides/virtual-tables.md).
+For more details see the [virtual-tables.md](virtual-tables.md).
 
 Below you can find a list of virtual tables. Sorted in alphabetical order (please keep it so when modifying!).
 

docs/guides/index.md

@@ -1,13 +0,0 @@
-# Guides 
-
-Instructions on how to build and test the Scylla codebase.
-
-## Contents
-
-```eval_rst
-.. toctree::
-   :glob:
-   :maxdepth: 1
-
-   *
-```

docs/index.md

@@ -1 +0,0 @@
-README.md

docs/source/conf.py

@@ -0,0 +1,141 @@
+# -*- coding: utf-8 -*-
+import os
+import sys
+import warnings
+from datetime import date
+
+from sphinx_scylladb_theme.utils import multiversion_regex_builder
+from recommonmark.transform import AutoStructify
+
+sys.path.insert(0, os.path.abspath(".."))
+
+# -- Global variables
+
+# Build documentation for the following tags and branches
+TAGS = []
+BRANCHES = []
+# Set the latest version.
+LATEST_VERSION = "branch-1.2"
+# Set which versions are not released yet.
+UNSTABLE_VERSIONS = ["master"]
+# Set which versions are deprecated
+DEPRECATED_VERSIONS = [""]
+
+# -- General configuration ------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings.
+extensions = [
+    "sphinx.ext.autodoc",
+    "sphinx.ext.todo",
+    "sphinx.ext.mathjax",
+    "sphinx.ext.githubpages",
+    "sphinx.ext.extlinks",
+    "sphinx_sitemap",
+    "sphinx_scylladb_theme",
+    "sphinx_multiversion",  # optional
+    "recommonmark",  # optional
+]
+
+# The suffix(es) of source filenames.
+source_suffix = [".rst", ".md"]
+
+# The master toctree document.
+master_doc = "index"
+
+# General information about the project.
+project = "ScyllaDB Documentation"
+copyright = str(date.today().year) + ", ScyllaDB. All rights reserved."
+author = u"Scylla Project Contributors"
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = "sphinx"
+
+# List of substitutions
+rst_prolog = """
+.. |rst| replace:: restructuredText
+"""
+# -- Options for not found extension ---------------------------------------
+
+# Template used to render the 404.html generated by this extension.
+notfound_template = "404.html"
+
+# Prefix added to all the URLs generated in the 404 page.
+notfound_urls_prefix = ""
+
+# -- Options for sitemap extension ---------------------------------------
+
+sitemap_url_scheme = "stable/{link}"
+
+# -- Options for redirect extension --------------------------------------
+
+# Read a YAML dictionary of redirections and generate an HTML file for each
+redirects_file = "_utils/redirections.yaml"
+
+# -- Options for multiversion extension ----------------------------------
+
+# Whitelist pattern for tags
+smv_tag_whitelist = multiversion_regex_builder(TAGS)
+# Whitelist pattern for branches
+smv_branch_whitelist = multiversion_regex_builder(BRANCHES)
+# Defines which version is considered to be the latest stable version.
+smv_latest_version = LATEST_VERSION
+# Defines the new name for the latest version.
+smv_rename_latest_version = "stable"
+# Whitelist pattern for remotes (set to None to use local branches only)
+smv_remote_whitelist = r"^origin$"
+# Pattern for released versions
+smv_released_pattern = r"^tags/.*$"
+# Format for versioned output directories inside the build directory
+smv_outputdir_format = "{ref.name}"
+
+# -- Options for HTML output ----------------------------------------------
+
+# The theme to use for pages.
+html_theme = "sphinx_scylladb_theme"
+html_theme_path = ["../.."]
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for the theme, see the
+# documentation.
+html_theme_options = {
+    "conf_py_path": "docs/source/",
+    "hide_edit_this_page_button": "false",
+    "hide_banner": "false",
+    "github_issues_repository": "scylladb/scylla",
+    "github_repository": "scylladb/scylla",
+    "hide_version_dropdown": ["master"],
+    "versions_unstable": UNSTABLE_VERSIONS,
+    "versions_deprecated": DEPRECATED_VERSIONS,
+}
+
+# Last updated format
+html_last_updated_fmt = "%d %b %Y"
+
+# Custom sidebar templates, maps document names to template names.
+html_sidebars = {"**": ["side-nav.html"]}
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = "ScyllaDocumentationdoc"
+
+# URL which points to the root of the HTML documentation.
+html_baseurl = "https://sphinx-theme.scylladb.com"
+
+# Dictionary of values to pass into the template engine’s context for all pages
+html_context = {"html_baseurl": html_baseurl}
+
+
+# -- Initialize Sphinx ----------------------------------------------
+def setup(sphinx):
+    warnings.filterwarnings(
+        action="ignore",
+        category=UserWarning,
+        message=r".*Container node skipped.*",
+    )
+    sphinx.add_config_value('recommonmark_config', {
+        'enable_eval_rst': True,
+    }, True)
+    sphinx.add_transform(AutoStructify)

docs/source/index.rst

@@ -0,0 +1,8 @@
+Welcome to ScyllaDB Documentation
+=================================
+
+.. toctree::
+  :glob:
+  :maxdepth: 1
+
+  alternator/alternator

gdbinit

@@ -1,5 +1,5 @@
 # Recommended .gdbinit for debugging scylla
-# See docs/guides/debugging.md for more information
+# See docs/dev/debugging.md for more information
 handle SIG34 pass noprint
 handle SIGUSR1 pass noprint
 set print pretty

idl/read_command.idl.hh

@@ -21,7 +21,7 @@ class specific_ranges {
 // format:
 // * legacy format
 // * native format
-// The wire format uses the legacy format. See docs/design-notes/reverse-reads.md
+// The wire format uses the legacy format. See docs/dev/reverse-reads.md
 // for more details on the formats.
 class partition_slice {
     std::vector<nonwrapping_range<clustering_key_prefix>> default_row_ranges();

query-request.hh

@@ -139,7 +139,7 @@ constexpr auto max_rows_if_set = std::numeric_limits<uint32_t>::max();
 // format:
 // * legacy format
 // * native format
-// The wire format uses the legacy format. See docs/design-notes/reverse-reads.md
+// The wire format uses the legacy format. See docs/dev/reverse-reads.md
 // for more details on the formats.
 class partition_slice {
     friend class ::partition_slice_builder;
@@ -249,7 +249,7 @@ public:
     friend std::ostream& operator<<(std::ostream& out, const specific_ranges& ps);
 };
 
-// See docs/design-notes/reverse-reads.md
+// See docs/dev/reverse-reads.md
 // In the following functions, `schema` may be reversed or not (both work).
 partition_slice legacy_reverse_slice_to_native_reverse_slice(const schema& schema, partition_slice slice);
 partition_slice native_reverse_slice_to_legacy_reverse_slice(const schema& schema, partition_slice slice);

range_tombstone.hh

@@ -180,7 +180,7 @@ public:
     }
 
     // Swap bounds to reverse range-tombstone -- as if it came from a table with
-    // reverse native order. See docs/design-notes/reverse-reads.md.
+    // reverse native order. See docs/dev/reverse-reads.md.
     void reverse() {
         std::swap(start, end);
         std::swap(start_kind, end_kind);

readers/mutation_source.hh

@@ -38,7 +38,7 @@ partition_presence_checker make_default_partition_presence_checker() {
 //
 // When reading in reverse, a reverse schema has to be passed (compared to the
 // table's schema), and a half-reverse (legacy) slice.
-// See docs/design-notes/reverse-reads.md for more details.
+// See docs/dev/reverse-reads.md for more details.
 // Partition-range forwarding is not yet supported in reverse mode.
 class mutation_source {
     using partition_range = const dht::partition_range&;

readers/reversing_v2.hh

@@ -27,7 +27,7 @@ namespace query {
 ///    account for the implicit null tombstone at the start of the stream moving
 ///    from start to end (due to reversing).
 /// Ordering of partitions themselves remains unchanged.
-/// For more details see docs/design-notes/reverse-reads.md.
+/// For more details see docs/dev/reverse-reads.md.
 ///
 /// The reader's schema (returned by `schema()`) is the reverse of `original`'s schema.
 ///

sstables/sstable_set.cc

@@ -488,7 +488,7 @@ class sstable_position_reader_queue : public position_reader_queue {
 public:
     // Assumes that `create_reader` returns readers that emit only fragments from partition `pk`.
     //
-    // For reversed reads `query_schema` must be reversed (see docs/design-notes/reverse-reads.md).
+    // For reversed reads `query_schema` must be reversed (see docs/dev/reverse-reads.md).
     sstable_position_reader_queue(const time_series_sstable_set& set,
             schema_ptr query_schema,
             std::function<flat_mutation_reader_v2(sstable&)> create_reader,

test/alternator/test_metrics.py

@@ -3,7 +3,7 @@
 # SPDX-License-Identifier: AGPL-3.0-or-later
 
 ##############################################################################
-# Tests for Scylla's metrics (see docs/design-notes/metrics.md) for Alternator
+# Tests for Scylla's metrics (see docs/dev/metrics.md) for Alternator
 # queries. Reproduces issue #9406, where although metrics was implemented for
 # Alternator requests, they were missing for some operations (BatchGetItem).
 # In the tests here we attempt to ensure that the metrics continue to work

tools/scylla-types.cc

@@ -204,7 +204,7 @@ without a leading 0x prefix, e.g. 00783562. For scylla-types to be able to
 examine the values, their type has to be provided. Types should be provided by
 their cassandra class names, e.g. org.apache.cassandra.db.marshal.Int32Type for
 the int32_type. The org.apache.cassandra.db.marshal. prefix can be ommited.
-See https://github.com/scylladb/scylla/blob/master/docs/design-notes/cql3-type-mapping.md
+See https://github.com/scylladb/scylla/blob/master/docs/dev/cql3-type-mapping.md
 for a mapping of cql3 types to Cassandra type class names.
 Compound types specify their subtypes inside () separated by comma, e.g.:
 MapType(Int32Type, BytesType). All provided values have to share the same type.