User GuideBenchmarking GuidecuVS Bench Tool

cuVS Bench Parameter Tuning Guide

View as Markdown

This guide outlines the various parameter settings that can be specified in cuVS Benchmarks yaml configuration files and explains the impact they have on corresponding algorithms to help inform their settings for benchmarking across desired levels of recall.

Benchmark modes

When you run benchmarks with BenchmarkOrchestrator.run_benchmark(), you can choose how parameters are explored:

Sweep mode (default)

Pass mode="sweep" or omit mode. The orchestrator builds the full Cartesian product of all build and search parameter lists defined in the algorithm YAML (see Creating and customizing algorithm configurations). Every valid combination (after constraint filtering) is run. Use this for exhaustive comparison across the configured parameter grid.

Tune mode

Pass mode="tune" to perform hyperparameter optimization using Optuna instead of running every combination. You must pass:

  • constraints (dict): The optimization target and optional bounds. One metric must be "maximize" or "minimize" (the goal). Others can set hard limits with {"min": X} or {"max": X}. Examples: {"recall": "maximize", "latency": {"max": 10}} or {"latency": "minimize", "recall": {"min": 0.95}}.
  • n_trials (int, optional): Maximum number of Optuna trials (default 100). Ignored in sweep mode.

Example:

1results = orchestrator.run_benchmark(
2    mode="tune",
3    dataset="deep-image-96-inner",
4    algorithms="cuvs_cagra",
5    constraints={"recall": "maximize", "latency": {"max": 5.0}},
6    n_trials=50,
7    count=10,
8    batch_size=10,
9)

The parameter tables below describe the build and search knobs that sweep mode varies and that tune mode can optimize.

cuVS Indexes

cuvs_brute_force

Use cuVS brute-force index for exact search. Brute-force has no further build or search parameters.

cuvs_ivf_flat

IVF-flat uses an inverted-file index, which partitions the vectors into a series of clusters, or lists, storing them in an interleaved format which is optimized for fast distance computation. The searching of an IVF-flat index reduces the total vectors in the index to those within some user-specified nearest clusters called probes.

IVF-flat is a simple algorithm which won’t save any space, but it provides competitive search times even at higher levels of recall.

ParameterTypeRequiredData TypeDefaultDescription
nlistbuildYPositive integer >01024Number of clusters to partition the vectors into. Larger values will put less points into each cluster but this will impact index build time as more clusters need to be trained.
niterbuildNPositive integer >020Number of kmeans iterations to use when training the ivf clusters
ratiobuildNPositive integer >021/ratio is the number of training points which should be used to train the clusters.
dataset_memory_typebuildN[device, host, mmap]mmapWhere should the dataset reside?
query_memory_typesearchN[device, host, mmap]deviceWhere should the queries reside?
nprobesearchYPositive integer >0The closest number of clusters to search for each query vector. Larger values will improve recall but will search more points in the index.

cuvs_ivf_pq

IVF-pq is an inverted-file index, which partitions the vectors into a series of clusters, or lists, in a similar way to IVF-flat above. The difference is that IVF-PQ uses product quantization to also compress the vectors, giving the index a smaller memory footprint. Unfortunately, higher levels of compression can also shrink recall, which a refinement step can improve when the original vectors are still available.

ParameterTypeRequiredData TypeDefaultDescription
nlistbuildYPositive integer >01024Number of clusters to partition the vectors into. Larger values will put less points into each cluster but this will impact index build time as more clusters need to be trained.
niterbuildNPositive integer >020Number of kmeans iterations to use when training the ivf clusters
ratiobuildNPositive integer >021/ratio is the number of training points which should be used to train the clusters.
pq_dimbuildNPositive integer. Multiple of 8.0Dimensionality of the vector after product quantization. When 0, a heuristic is used to select this value.
pq_bitsbuildNPositive integer [4-8]8Bit length of the vector element after quantization.
codebook_kindbuildN[cluster, subspace]subspaceType of codebook. See IVF-PQ index overview for more detail
dataset_memory_typebuildN[device, host, mmap]mmapWhere should the dataset reside?
query_memory_typesearchN[device, host, mmap]deviceWhere should the queries reside?
nprobesearchYPositive integer >020The closest number of clusters to search for each query vector. Larger values will improve recall but will search more points in the index.
internalDistanceDtypesearchN[float, half]halfThe precision to use for the distance computations. Lower precision can increase performance at the cost of accuracy.
smemLutDtypesearchN[float, half, fp8]halfThe precision to use for the lookup table in shared memory. Lower precision can increase performance at the cost of accuracy.
refine_ratiosearchNPositive integer >01refine_ratio * k nearest neighbors are queried from the index initially and an additional refinement step improves recall by selecting only the best k neighbors.

cuvs_cagra

CAGRA uses a graph-based index, which creates an intermediate, approximate kNN graph using IVF-PQ and then further refining and optimizing to create a final kNN graph. This kNN graph is used by CAGRA as an index for search.

ParameterTypeRequiredData TypeDefaultDescription
graph_degreebuildNPositive integer >064Degree of the final kNN graph index.
intermediate_graph_degreebuildNPositive integer >0128Degree of the intermediate kNN graph before the CAGRA graph is optimized
graph_build_algobuildN[IVF_PQ, NN_DESCENT, ACE]IVF_PQAlgorithm to use for building the initial kNN graph, from which CAGRA will optimize into the navigable CAGRA graph
dataset_memory_typebuildN[device, host, mmap]mmapWhere should the dataset reside?
npartitionsbuildNPositive integer >01The number of partitions to use for the ACE build. Small values might improve recall but potentially degrade performance and increase memory usage. Partitions should not be too small to prevent issues in KNN graph construction. The partition size is on average 2 * (n_rows / npartitions) * dim * sizeof(T). 2 is because of the core and augmented vectors. Please account for imbalance in the partition sizes (up to 3x in our tests).
build_dirbuildNString”/tmp/ace_build”The directory to use for the ACE build. Must be specified when using ACE build. This should be the fastest disk in the system and hold enough space for twice the dataset, final graph, and label mapping.
ef_constructionbuildYPositive integer >0120Controls index time and accuracy when using ACE build. Bigger values increase the index quality. At some point, increasing this will no longer improve the quality.
use_diskbuildNBooleanfalseWhether to use disk-based storage for ACE build. When true, forces ACE to use disk-based storage even if the graph fits in host and GPU memory. When false, ACE will use in-memory storage if the graph fits in host and GPU memory and disk-based storage otherwise.
query_memory_typesearchN[device, host, mmap]deviceWhere should the queries reside?
itopksearchNPositive integer >064Number of intermediate search results retained during the search. Higher values improve search accuracy at the cost of speed
search_widthsearchNPositive integer >01Number of graph nodes to select as the starting point for the search in each iteration.
max_iterationssearchNPositive integer >=00Upper limit of search iterations. Auto select when 0
algosearchN[auto, single_cta, multi_cta, multi_kernel]autoAlgorithm to use for search. It’s usually best to leave this to auto.
persistentsearchNBooleanfalseEnables the persistent CAGRA search kernel for high-throughput search with many concurrent client threads. Persistent search currently requires single_cta; auto selects it when persistent mode is enabled.
persistent_lifetimesearchNPositive float >02Seconds the persistent kernel waits for additional work before stopping. Increase this if searches arrive with gaps between batches.
persistent_device_usagesearchNFloat in (0, 1]1.0Fraction of the maximum grid size used by the persistent kernel. Lower values can leave GPU resources available for other work.
graph_memory_typesearchN[device, host_pinned, host_huge_page]deviceMemory type to store graph
internal_dataset_memory_typesearchN[device, host_pinned, host_huge_page]deviceMemory type to store dataset

The graph_memory_type or internal_dataset_memory_type options can be useful for large datasets that do not fit the device memory. Setting internal_dataset_memory_type other than device has negative impact on search speed. Using host_huge_page option is only supported on systems with Heterogeneous Memory Management or on platforms that natively support GPU access to system allocated memory, for example Grace Hopper.

To fine tune CAGRA index building we can customize IVF-PQ index builder options using the following settings. These take effect only if graph_build_algo == "IVF_PQ". It is recommended to experiment using a separate IVF-PQ index to find the config that gives the largest QPS for large batch. Recall does not need to be very high, since CAGRA further optimizes the kNN neighbor graph. Some of the default values are derived from the dataset size which is assumed to be [n_vecs, dim].

ParameterTypeRequiredData TypeDefaultDescription
ivf_pq_build_nlistbuildNPositive integer >0sqrt(n_vecs)Number of clusters to partition the vectors into. Larger values will put less points into each cluster but this will impact index build time as more clusters need to be trained.
ivf_pq_build_niterbuildNPositive integer >025Number of k-means iterations to use when training the clusters.
ivf_pq_build_ratiobuildNPositive integer >0101/ratio is the number of training points which should be used to train the clusters.
ivf_pq_pq_dimbuildNPositive integer. Multiple of 8dim/2 rounded up to 8Dimensionality of the vector after product quantization. When 0, a heuristic is used to select this value. pq_dim * pq_bits must be a multiple of 8.
ivf_pq_build_pq_bitsbuildNPositive integer [4-8]8Bit length of the vector element after quantization.
ivf_pq_build_codebook_kindbuildN[cluster, subspace]subspaceType of codebook. See IVF-PQ index overview for more detail
ivf_pq_build_nprobesearchNPositive integer >0min(2*dim, nlist)The closest number of clusters to search for each query vector. Larger values will improve recall but will search more points in the index.
ivf_pq_build_internalDistanceDtypesearchN[float, half]halfThe precision to use for the distance computations. Lower precision can increase performance at the cost of accuracy.
ivf_pq_build_smemLutDtypesearchN[float, half, fp8]fp8The precision to use for the lookup table in shared memory. Lower precision can increase performance at the cost of accuracy.
ivf_pq_build_refine_ratiosearchNPositive integer >02refine_ratio * k nearest neighbors are queried from the index initially and an additional refinement step improves recall by selecting only the best k neighbors.

Alternatively, if graph_build_algo == "NN_DESCENT", then we can customize the following parameters

ParameterTypeRequiredData TypeDefaultDescription
nn_descent_niterbuildNPositive integer >020Number of nn-descent iterations
nn_descent_intermediate_graph_degreebuildNPositive integer >0cagra.intermediate_graph_degree * 1.5Intermadiate graph degree during nn-descent iterations
nn_descent_termination_thresholdbuildNPositive float >01e-4Early stopping threshold for nn-descent convergence

cuvs_cagra_hnswlib

This is a benchmark that enables interoperability between CAGRA built HNSW search. It uses the CAGRA built graph as the base layer of an hnswlib index to search queries only within the base layer (this is enabled with a simple patch to hnswlib).

build : Same as build of CAGRA

search : Same as search of Hnswlib

cuvs_vamana

Benchmark for building an in-memory Vamana graph based index on the GPU and interoperability with DiskANN for search.

ParameterTypeRequiredData TypeDefaultDescription
graph_degreebuildNPositive integer >032Maximum degree of the graph index
visited_sizebuildNPositive integer >064Maximum number of visited nodes per search corresponds to the L parameter in the Vamana literature
alphabuildNPositive float >01.2Alpha for pruning parameter
L_searchsearchYPositive integer >0Maximum number of visited nodes per search corresponds to the L parameter in the Vamana literature. Larger values improve recall at the cost of search time.

FAISS Indexes

faiss_gpu_flat

Use FAISS flat index on the GPU, which performs an exact search using brute-force and doesn’t have any further build or search parameters.

faiss_gpu_ivf_flat

IVF-flat uses an inverted-file index, which partitions the vectors into a series of clusters, or lists, storing them in an interleaved format which is optimized for fast distance computation. The searching of an IVF-flat index reduces the total vectors in the index to those within some user-specified nearest clusters called probes.

IVF-flat is a simple algorithm which won’t save any space, but it provides competitive search times even at higher levels of recall.

ParameterTypeRequiredData TypeDefaultDescription
nlistsbuildYPositive integer >0Number of clusters to partition the vectors into. Larger values will put less points into each cluster but this will impact index build time as more clusters need to be trained
ratiobuildNPositive integer >021/ratio is the number of training points which should be used to train the clusters.
nprobesearchYPositive integer >020The closest number of clusters to search for each query vector. Larger values will improve recall but will search more points in the index.

faiss_gpu_ivf_pq

IVF-pq is an inverted-file index, which partitions the vectors into a series of clusters, or lists, in a similar way to IVF-flat above. The difference is that IVF-PQ uses product quantization to also compress the vectors, giving the index a smaller memory footprint. Unfortunately, higher levels of compression can also shrink recall, which a refinement step can improve when the original vectors are still available.

ParameterTypeRequiredData TypeDefaultDescription
nlistbuildYPositive integer >0Number of clusters to partition the vectors into. Larger values will put less points into each cluster but this will impact index build time as more clusters need to be trained.
ratiobuildNPositive integer >021/ratio is the number of training points which should be used to train the clusters.
M_ratiobuildYPositive integer. Power of 2 [8-64]Ratio of number of chunks or subquantizers for each vector. Computed by dims / M_ratio
usePrecomputedbuildNBooleanfalseUse pre-computed lookup tables to speed up search at the cost of increased memory usage.
useFloat16buildNBooleanfalseUse half-precision floats for clustering step.
nprobesearchYPositive integer >0The closest number of clusters to search for each query vector. Larger values will improve recall but will search more points in the index.
refine_ratiosearchNPositive number >=11refine_ratio * k nearest neighbors are queried from the index initially and an additional refinement step improves recall by selecting only the best k neighbors.

faiss_cpu_flat

Use FAISS flat index on the CPU, which performs an exact search using brute-force and doesn’t have any further build or search parameters.

ParameterTypeRequiredData TypeDefaultDescription
numThreadssearchNPositive integer >01Number of threads to use for queries.

faiss_cpu_ivf_flat

Use FAISS IVF-Flat index on CPU

ParameterTypeRequiredData TypeDefaultDescription
nlistsbuildYPositive integer >0Number of clusters to partition the vectors into. Larger values will put less points into each cluster but this will impact index build time as more clusters need to be trained
ratiobuildNPositive integer >021/ratio is the number of training points which should be used to train the clusters.
nprobesearchYPositive integer >0The closest number of clusters to search for each query vector. Larger values will improve recall but will search more points in the index.
numThreadssearchNPositive integer >01Number of threads to use for queries.

faiss_cpu_ivf_pq

Use FAISS IVF-PQ index on CPU

ParameterTypeRequiredData TypeDefaultDescription
nlistbuildYPositive integer >0Number of clusters to partition the vectors into. Larger values will put less points into each cluster but this will impact index build time as more clusters need to be trained.
ratiobuildNPositive integer >021/ratio is the number of training points which should be used to train the clusters.
MbuildYPositive integer. Power of 2 [8-64]Ratio of number of chunks or subquantizers for each vector. Computed by dims / M_ratio
usePrecomputedbuildNBooleanfalseUse pre-computed lookup tables to speed up search at the cost of increased memory usage.
bitsPerCodebuildNPositive integer [4-8]8Number of bits for representing each quantized code.
nprobesearchYPositive integer >0The closest number of clusters to search for each query vector. Larger values will improve recall but will search more points in the index.
refine_ratiosearchNPositive number >=11refine_ratio * k nearest neighbors are queried from the index initially and an additional refinement step improves recall by selecting only the best k neighbors.
numThreadssearchNPositive integer >01Number of threads to use for queries.

HNSW

cuvs_hnsw

cuVS HNSW builds an HNSW index using the ACE (Augmented Core Extraction) algorithm, which enables GPU-accelerated HNSW index construction for datasets too large to fit in GPU memory.

ParameterTypeRequiredData TypeDefaultDescription
hierarchybuildN[NONE, CPU, GPU]NONEType of HNSW hierarchy to build. NONE creates a base-layer-only index, CPU builds full hierarchy on CPU, GPU builds full hierarchy on GPU.
efConstructionbuildYPositive integer >0Controls index time and accuracy. Bigger values increase the index quality. At some point, increasing this will no longer improve the quality.
MbuildYPositive integer. Often between 2-100Number of bi-directional links create for every new element during construction. Higher values work for higher intrinsic dimensionality and/or high recall, low values can work for datasets with low intrinsic dimensionality and/or low recalls. Also affects the algorithm’s memory consumption.
numThreadsbuildNPositive integer >01Number of threads to use to build the index.
npartitionsbuildNPositive integer >01Number of partitions to use for the ACE build. Small values might improve recall but potentially degrade performance and increase memory usage. The partition size is on average 2 * (n_rows / npartitions) * dim * sizeof(T). 2 is because of the core and augmented vectors. Please account for imbalance in the partition sizes (up to 3x in our tests).
ef_constructionbuildNPositive integer >0120Controls index time and accuracy when using ACE build. Bigger values increase the index quality. At some point, increasing this will no longer improve the quality.
build_dirbuildNString”/tmp/ace_build”The directory to use for the ACE build. This should be the fastest disk in the system and hold enough space for twice the dataset, final graph, and label mapping.
use_diskbuildNBooleanfalseWhether to use disk-based storage for ACE build. When true, forces ACE to use disk-based storage even if the graph fits in host and GPU memory. When false, ACE will use in-memory storage if the graph fits in host and GPU memory and disk-based storage otherwise.
efsearchYPositive integer >0Size of the dynamic list for the nearest neighbors used for search. Higher value leads to more accurate but slower search. Cannot be lower than k.
numThreadssearchNPositive integer >01Number of threads to use for queries.

hnswlib

ParameterTypeRequiredData TypeDefaultDescription
efConstructionbuildYPositive integer >0Controls index time and accuracy. Bigger values increase the index quality. At some point, increasing this will no longer improve the quality.
MbuildYPositive integer. Often between 2-100Number of bi-directional links create for every new element during construction. Higher values work for higher intrinsic dimensionality and/or high recall, low values can work for datasets with low intrinsic dimensionality and/or low recalls. Also affects the algorithm’s memory consumption.
numThreadsbuildNPositive integer >01Number of threads to use to build the index.
efsearchYPositive integer >0Size of the dynamic list for the nearest neighbors used for search. Higher value leads to more accurate but slower search. Cannot be lower than k.
numThreadssearchNPositive integer >01Number of threads to use for queries.

Please refer to HNSW algorithm parameters guide from hnswlib to learn more about these arguments.

DiskANN

diskann_memory

Use DiskANN in-memory index for approximate search.

ParameterTypeRequiredData TypeDefaultDescription
RbuildYPositive integer >0Maximum degree of the graph index
L_buildbuildYPositive integer >0number of visited nodes per greedy search during graph construction
alphabuildNPositive number >=11.2controls the pruning parameter of the graph construction
num_threadsbuildNPositive integer >0omp_get_max_threads()Number of CPU threads to use to build the index.
L_searchsearchYPositive integer >0visited list size during search