User GuideBenchmarking GuidecuVS Bench Tool

cuVS Bench Backends

View as Markdown

This page explains how cuVS Bench separates benchmark orchestration from the system that actually builds and searches an index. Use it when you want to understand the built-in C++ benchmark backend, add a new backend for another product or service, or add a new indexing algorithm to the existing C++ backend.

cuVS Bench uses two pieces for each backend:

PiecePurpose
Config loaderReads user inputs and configuration files, expands parameter sweeps, and returns the datasets and index configurations to run.
BackendUses those configurations to build indexes, run searches, and return benchmark results.

Both pieces are registered under the same backend type name. The default backend type is cpp_gbench, which runs the C++ Google Benchmark executables.

1from cuvs_bench.orchestrator import BenchmarkOrchestrator
2
3orchestrator = BenchmarkOrchestrator(backend_type="cpp_gbench")
4results = orchestrator.run_benchmark(
5    dataset="deep-image-96-inner",
6    algorithms="cuvs_cagra",
7    count=10,
8    batch_size=10,
9    build=True,
10    search=True,
11)

How a benchmark run works

  1. The user calls BenchmarkOrchestrator(...).run_benchmark(...).
  2. The orchestrator finds the config loader registered for the requested backend type.
  3. The config loader returns a DatasetConfig and one or more BenchmarkConfig objects.
  4. The orchestrator creates the backend registered for the same backend type.
  5. The backend runs build(...) and search(...), then returns BuildResult and SearchResult objects.

The config loader decides what to run. The backend decides how to run it.

Configuration contract

A config loader receives the arguments passed to run_benchmark(), such as dataset, dataset_path, algorithms, count, batch_size, groups, and backend-specific options. It returns:

Return valuePurpose
DatasetConfigDataset metadata, including vector files, ground-truth files, distance metric, dimensions, and optional subset size.
List[BenchmarkConfig]One or more benchmark configurations. Each contains index configurations and backend-specific options.

Each IndexConfig describes one index to benchmark:

FieldPurpose
nameHuman-readable index name, usually including parameter values.
algoAlgorithm name.
build_paramBuild parameters for the index.
search_paramsSearch parameter combinations to benchmark.
filePath or identifier where the backend stores the index.

The following minimal loader creates one dataset, one index, and one search configuration:

1from cuvs_bench.orchestrator.config_loaders import (
2    ConfigLoader,
3    DatasetConfig,
4    BenchmarkConfig,
5    IndexConfig,
6)
7
8class MyConfigLoader(ConfigLoader):
9    @property
10    def backend_type(self) -> str:
11        return "my_backend"
12
13    def load(self, dataset, dataset_path, algorithms, count=10, batch_size=10000, **kwargs):
14        dataset_config = DatasetConfig(
15            name=dataset,
16            base_file=...,
17            query_file=...,
18            groundtruth_neighbors_file=...,
19            distance="euclidean",
20            dims=128,
21        )
22        index = IndexConfig(
23            name=f"{algorithms}.default",
24            algo=algorithms,
25            build_param={"nlist": 1024},
26            search_params=[{"nprobe": 10}],
27            file=...,
28        )
29        benchmark_config = BenchmarkConfig(
30            indexes=[index],
31            backend_config={
32                "host": ...,
33                "port": ...,
34                "index_name": ...,
35            },
36        )
37        return dataset_config, [benchmark_config]

Adding a backend

Add a new backend when cuVS Bench needs to drive a different execution path, such as a vector database, remote service, or custom benchmark runner.

  1. Implement a config loader by subclassing ConfigLoader from cuvs_bench.orchestrator.config_loaders. Its load() method should return (DatasetConfig, List[BenchmarkConfig]).
  2. Implement a backend by subclassing BenchmarkBackend from cuvs_bench.backends.base. Its build() method should return BuildResult; its search() method should return SearchResult.
  3. Register both pieces with the same backend type name.
  4. Run benchmarks with BenchmarkOrchestrator(backend_type="my_backend").
1from cuvs_bench.orchestrator import register_config_loader
2from cuvs_bench.backends import get_registry
3
4register_config_loader("my_backend", MyConfigLoader)
5get_registry().register("my_backend", MyBackend)

Example: Elasticsearch backend

This example shows the shape of a network backend. The loader creates the dataset and benchmark configs. The backend uses backend_config to connect to the service, build the index, run search, and return cuVS Bench result objects.

1from cuvs_bench.orchestrator.config_loaders import (
2    ConfigLoader,
3    DatasetConfig,
4    BenchmarkConfig,
5    IndexConfig,
6)
7
8class ElasticsearchConfigLoader(ConfigLoader):
9    @property
10    def backend_type(self) -> str:
11        return "elasticsearch"
12
13    def load(self, dataset, dataset_path, algorithms, count=10, batch_size=10000, **kwargs):
14        dataset_config = DatasetConfig(
15            name=dataset,
16            base_file=...,
17            query_file=...,
18            groundtruth_neighbors_file=...,
19            distance="euclidean",
20            dims=kwargs.get("dims", 128),
21        )
22        index = IndexConfig(
23            name=f"{algorithms}.es",
24            algo=algorithms,
25            build_param={},
26            search_params=[{"ef_search": 100}],
27            file=...,
28        )
29        benchmark_config = BenchmarkConfig(
30            indexes=[index],
31            backend_config={
32                "host": ...,
33                "port": ...,
34                "index_name": ...,
35                "algo": algorithms,
36            },
37        )
38        return dataset_config, [benchmark_config]
1import numpy as np
2from cuvs_bench.backends.base import (
3    BenchmarkBackend,
4    BuildResult,
5    SearchResult,
6)
7
8class ElasticsearchBackend(BenchmarkBackend):
9    @property
10    def algo(self) -> str:
11        return self.config.get("algo", "elasticsearch")
12
13    def build(self, dataset, indexes, force=False, dry_run=False):
14        return BuildResult(
15            index_path=indexes[0].file if indexes else "",
16            build_time_seconds=0.0,
17            index_size_bytes=0,
18            algorithm=self.algo,
19            build_params=indexes[0].build_param if indexes else {},
20            metadata={},
21            success=True,
22        )
23
24    def search(
25        self,
26        dataset,
27        indexes,
28        k,
29        batch_size=10000,
30        mode="latency",
31        force=False,
32        search_threads=None,
33        dry_run=False,
34    ):
35        n_queries = dataset.n_queries
36        return SearchResult(
37            neighbors=np.zeros((n_queries, k), dtype=np.int64),
38            distances=np.zeros((n_queries, k), dtype=np.float32),
39            search_time_ms=0.0,
40            queries_per_second=0.0,
41            recall=0.0,
42            algorithm=self.algo,
43            search_params=indexes[0].search_params if indexes else [],
44            success=True,
45        )
1from cuvs_bench.orchestrator import register_config_loader
2from cuvs_bench.backends import get_registry
3
4register_config_loader("elasticsearch", ElasticsearchConfigLoader)
5get_registry().register("elasticsearch", ElasticsearchBackend)

Components at a glance

ComponentDescription
ConfigLoaderAbstract class whose load(**kwargs) method returns (DatasetConfig, List[BenchmarkConfig]). Register with register_config_loader(backend_type, loader_class).
BenchmarkBackendAbstract class whose build(...) method returns BuildResult and whose search(...) method returns SearchResult. Register with BackendRegistry.register(name, backend_class).
BackendRegistrySingleton registry returned by get_registry(). It maps backend type names to backend classes.

C++ Backend

The built-in CppGoogleBenchmarkBackend uses backend_type="cpp_gbench". Its config loader reads YAML under config/datasets and config/algos, expands parameter combinations, and validates constraints. Its backend runs the C++ benchmark executables and merges their results.

Adding a new C++ algorithm usually means adding another executable and YAML config for this backend. It does not require a new backend type.

Implementation and configuration

New algorithms should be C++ classes that inherit class ANN from cpp/bench/ann/src/ann.h and implement all pure virtual functions.

Define separate build and search parameter structs. The search parameter struct should inherit struct ANN<T>::AnnSearchParam.

1template<typename T>
2class HnswLib : public ANN<T> {
3public:
4  struct BuildParam {
5    int M;
6    int ef_construction;
7    int num_threads;
8  };
9
10  using typename ANN<T>::AnnSearchParam;
11  struct SearchParam : public AnnSearchParam {
12    int ef;
13    int num_threads;
14  };
15
16  // ...
17};

The benchmark program consumes generated JSON files for indexes, build parameters, and search parameters. The JSON objects map to YAML build_param objects and search_param arrays.

1{
2  "name": "hnswlib.M12.ef500.th32",
3  "algo": "hnswlib",
4  "build_param": {"M": 12, "efConstruction": 500, "numThreads": 32},
5  "file": "/path/to/file",
6  "search_params": [
7    {"ef": 10, "numThreads": 1},
8    {"ef": 20, "numThreads": 1},
9    {"ef": 40, "numThreads": 1}
10  ],
11  "search_result_file": "/path/to/file"
12}

Parse build and search parameters from JSON:

1template<typename T>
2void parse_build_param(const nlohmann::json& conf,
3                       typename cuann::HnswLib<T>::BuildParam& param) {
4  param.ef_construction = conf.at("efConstruction");
5  param.M = conf.at("M");
6  if (conf.contains("numThreads")) {
7    param.num_threads = conf.at("numThreads");
8  }
9}
10
11template<typename T>
12void parse_search_param(const nlohmann::json& conf,
13                        typename cuann::HnswLib<T>::SearchParam& param) {
14  param.ef = conf.at("ef");
15  if (conf.contains("numThreads")) {
16    param.num_threads = conf.at("numThreads");
17  }
18}

Add matching if cases to create_algo() and create_search_param() in cpp/bench/ann/. The string literal must match the algo value in the configuration file.

1if (algo == "hnswlib") {
2   // ...
3}

Adding a CMake target

cuvs/cpp/bench/ann/CMakeLists.txt provides a CMake helper for new benchmark targets:

1ConfigureAnnBench(
2  NAME <algo_name>
3  PATH </path/to/algo/benchmark/source/file>
4  INCLUDES <additional_include_directories>
5  CXXFLAGS <additional_cxx_flags>
6  LINKS <additional_link_library_targets>
7)

Example target for HNSWLIB:

1ConfigureAnnBench(
2  NAME HNSWLIB PATH bench/ann/src/hnswlib/hnswlib_benchmark.cpp INCLUDES
3  ${CMAKE_CURRENT_BINARY_DIR}/_deps/hnswlib-src/hnswlib CXXFLAGS "${HNSW_CXX_FLAGS}"
4)

This creates HNSWLIB_ANN_BENCH, which runs HNSWLIB benchmarks.

Add an algos.yaml entry that maps the algorithm name to its executable and declares whether the algorithm requires a GPU:

1cuvs_ivf_pq:
2  executable: CUVS_IVF_PQ_ANN_BENCH
3  requires_gpu: true

executable specifies the binary used to build and search the index. cuVS Bench expects it to be available in cuvs/cpp/build/. requires_gpu tells cuVS Bench whether the algorithm must run on a GPU node.

Summary

cuVS Bench backends let the same benchmark workflow run against different execution targets. A config loader describes the dataset and parameter combinations, while a backend performs the build and search work. Use a new backend type for a new execution environment, and use the existing C++ backend when you are only adding another C++ ANN benchmark executable.