KVBM Configuration Reference | NVIDIA Dynamo Documentation

The KV Block Manager (KVBM) is configured through two layers:

DYN_KVBM_* runtime environment variables — the primary, user-facing controls for cache tier sizing, offload behavior, object storage, distributed (leader/worker) mode, and metrics. Set these in the worker container env. This is what most deployments use.
The KVBM_* / TOML figment system — a lower-level, layered configuration for the KVBM runtime (Tokio/Rayon threads, messenger transport, NixL backends, offload policies, event publishing). Used for advanced tuning and for vLLM’s kv_connector_extra_config.

For the conceptual model and worked examples, see the KVBM Guide. Names and defaults on this page come from environment_names.rs and the kvbm-config crate.

Cache tiers

KVBM offloads KV blocks GPU (G1) → host/CPU (G2) → disk (G3) → object storage (G4). Enable a tier by giving it a size; each tier accepts either a size in GB or an explicit block count (the block-count override wins).

Set the CPU cache larger than the engine’s GPU KV cache — a good rule of thumb is DYN_KVBM_CPU_CACHE_GB >= 100 and DYN_KVBM_DISK_CACHE_GB >= DYN_KVBM_CPU_CACHE_GB. If the CPU cache is smaller than the device cache, KVBM churns (offloading blocks after every forward pass) and you may see performance degradation. Consult your engine’s KV-cache configuration for the minimum useful value.

DYN_KVBM_CPU_CACHE_GB

integerDefaults to unset

Size of the host (G2, pinned CPU memory) cache in GB. Setting this enables CPU offload.

DYN_KVBM_CPU_CACHE_OVERRIDE_NUM_BLOCKS

integerDefaults to unset

Host cache size as an explicit block count. Takes priority over DYN_KVBM_CPU_CACHE_GB.

DYN_KVBM_DISK_CACHE_GB

integerDefaults to unset

Size of the disk (G3) cache in GB. Setting this enables disk offload.

DYN_KVBM_DISK_CACHE_OVERRIDE_NUM_BLOCKS

integerDefaults to unset

Disk cache size as an explicit block count. Takes priority over DYN_KVBM_DISK_CACHE_GB.

DYN_KVBM_DISK_CACHE_DIR

stringDefaults to unset

Directory for disk cache files. When unset, a default path is used.

Offload and transfer

DYN_KVBM_DISABLE_DISK_OFFLOAD_FILTER

booleanDefaults to false

Disable the disk offload filter, allowing all eligible blocks to offload to disk. The filter is not supported in disk-cache-only mode.

DYN_KVBM_HOST_OFFLOAD_PREFIX_MIN_PRIORITY

integerDefaults to unset

Minimum prefix priority a block must have to be offloaded to the host tier. Raises the bar for what gets offloaded.

DYN_KVBM_TRANSFER_BATCH_SIZE

integerDefaults to unset

Maximum number of blocks per transfer batch.

DYN_KVBM_MAX_TRANSFER_BATCH_SIZE

integerDefaults to unset

Upper bound on the transfer batch size.

DYN_KVBM_MAX_CONCURRENT_TRANSFERS

integerDefaults to unset

Maximum number of block transfers in flight at once.

DYN_KVBM_DISABLE_WRITE_COMBINED

booleanDefaults to false

Disable write-combined memory for transfers.

Disk backend

DYN_KVBM_DISK_ALLOCATOR_TYPE

stringDefaults to unset

Disk allocator implementation. open-direct uses O_DIRECT I/O.

DYN_KVBM_DISK_DISABLE_O_DIRECT

booleanDefaults to false

Disable O_DIRECT on disk I/O. Use on filesystems that do not support it.

DYN_KVBM_DISK_ZEROFILL_FALLBACK

booleanDefaults to false

Fall back to zero-filling when preallocating disk cache files is not supported.

Object storage (G4)

The G4 tier stores blocks in an S3-compatible object store. Enable it with DYN_KVBM_OBJECT_ENABLED=1 and supply the bucket and credentials.

DYN_KVBM_OBJECT_ENABLED

booleanDefaults to false

Enable object storage. Set to 1 to enable.

DYN_KVBM_OBJECT_BUCKET

stringDefaults to unset

Bucket name for the object-storage cache. Supports a {worker_id} template for per-worker buckets, for example kv-cache-{worker_id}.

DYN_KVBM_OBJECT_ENDPOINT

stringDefaults to unset

Endpoint URL for the object store. Set for S3-compatible services such as MinIO.

DYN_KVBM_OBJECT_REGION

stringDefaults to unset

Region for the object store.

DYN_KVBM_OBJECT_ACCESS_KEY

stringDefaults to unset

Access key for object-store authentication.

DYN_KVBM_OBJECT_SECRET_KEY

stringDefaults to unset

Secret key for object-store authentication.

DYN_KVBM_OBJECT_NUM_BLOCKS

integerDefaults to unset

Number of blocks to store in object storage.

Distributed (leader/worker)

In multi-worker deployments, one KVBM leader coordinates the worker cache tiers over ZMQ.

DYN_KVBM_LEADER_WORKER_INIT_TIMEOUT_SECS

integerDefaults to unset

Timeout in seconds for KVBM leader and worker initialization. Raise it for large multi-node deployments that take longer to come up.

DYN_KVBM_LEADER_ZMQ_HOST

stringDefaults to unset

ZMQ host for the KVBM leader.

DYN_KVBM_LEADER_ZMQ_PUB_PORT

integerDefaults to 56001

ZMQ publish port for the KVBM leader. The TRT-LLM integration derives its port from this value.

DYN_KVBM_LEADER_ZMQ_ACK_PORT

stringDefaults to unset

ZMQ acknowledgment port for the KVBM leader.

MLA and NCCL

DYN_KVBM_NCCL_MLA_MODE

booleanDefaults to false

Enable the NCCL broadcast optimization for Multi-head Latent Attention (MLA) models. When disabled, each GPU loads KV blocks independently. Set true when running MLA models with KVBM.

DYN_KVBM_NCCL_MAX_CTAS

integerDefaults to unset

Maximum number of CTAs (cooperative thread arrays) NCCL may use for KVBM broadcasts.

DYN_KVBM_ENABLE_NUMA

booleanDefaults to false

Enable NUMA-aware allocation for the host cache.

NixL backends

Enable individual NixL transfer backends with the DYN_KVBM_NIXL_BACKEND_<backend> pattern.

DYN_KVBM_NIXL_BACKEND_<backend>

booleanDefaults to unset

Enable or disable a specific NixL backend, for example DYN_KVBM_NIXL_BACKEND_UCX=true or DYN_KVBM_NIXL_BACKEND_GDS=true. Backend names are case-insensitive.

Metrics and debugging

DYN_KVBM_METRICS

booleanDefaults to false

Enable the KVBM metrics endpoint.

DYN_KVBM_METRICS_PORT

integerDefaults to unset

Port for the KVBM metrics endpoint.

DYN_KVBM_ENABLE_RECORD

booleanDefaults to false

Enable KVBM event recording for debugging.

DYN_KVBM_CACHE_STATS_LOG_INTERVAL_SECS

integerDefaults to unset

Interval in seconds between cache-statistics log lines.

DYN_KVBM_CACHE_STATS_MAX_REQUESTS

integerDefaults to unset

Maximum number of requests to accumulate in cache statistics.

Advanced: the `KVBM_*` / TOML system

Beyond the DYN_KVBM_* variables above, the KVBM runtime (crate kvbm-config) has a layered configuration system for low-level runtime tuning. Most deployments do not need it.

How it loads

Sources are merged in priority order (lowest to highest):

Code defaults
System file /opt/dynamo/etc/kvbm.toml
TOML file at the path in KVBM_CONFIG_PATH
KVBM_* environment variables
Programmatic JSON overrides (for example, vLLM’s kv_connector_extra_config)

Leader and worker roles can carry different values via figment profiles (profile.leader.*, profile.worker.* in TOML/JSON).

KVBM_CONFIG_PATH

stringDefaults to unset

Path to a TOML config file merged above the system file and below KVBM_* env vars.

Environment-variable groups

Each config section maps to an env prefix. The suffix after the prefix is lowercased into the field name (for example, KVBM_TOKIO_WORKER_THREADS → tokio.worker_threads).

Env prefix	Section	Key fields (default)
`KVBM_TOKIO_`	Async runtime	`WORKER_THREADS` (1), `MAX_BLOCKING_THREADS` (Tokio default 512)
`KVBM_RAYON_`	CPU thread pool	`NUM_THREADS` (logical CPUs)
`KVBM_MESSENGER_BACKEND_`	TCP transport	`TCP_ADDR`, `TCP_INTERFACE`, `TCP_PORT` (0 = OS-assigned)
`KVBM_MESSENGER_DISCOVERY_`	Peer discovery	`CLUSTER_ID`, plus etcd/p2p/filesystem fields
`KVBM_NIXL_`	NixL backends	`BACKENDS` (default `UCX`, `POSIX`)
`KVBM_CACHE_HOST_`	Host tier	`SIZE_GB`, `NUM_BLOCKS`
`KVBM_CACHE_DISK_`	Disk tier	`SIZE_GB`, `NUM_BLOCKS`, `USE_GDS` (false), `STORAGE_PATH`
`KVBM_CACHE_PARALLELISM`	Parallelism mode	`tensor_parallel` (default) or `replicated_data`
`KVBM_EVENTS_`	Event publishing	`ENABLED` (false), `SUBJECT` (`kvbm.events`), `CHANNEL_CAPACITY` (1024), `POLICY` (`power_of_two`)
`KVBM_EVENTS_BATCHING_`	Event batching	`WINDOW_DURATION_MS` (10), `MAX_BATCH_SIZE` (1024)

Nested sections without an env prefix (offload policies, onboard mode, object storage) are set through the TOML file or JSON overrides:

Offload policies (offload.g1_to_g2, offload.g2_to_g3): a policies list of pass_all / presence / presence_lfu, evaluated with AND logic. presence_lfu.min_lfu_count defaults to 8.
Onboard mode (onboard.mode): inter (async, default) or intra (synchronous layer-wise).
Object storage (object.client): s3 or nixl client; S3 fields bucket (kvbm-blocks), region (us-east-1), endpoint_url, force_path_style, max_concurrent_requests (16).

TOML example

1 [tokio]
2 worker_threads = 4
3 
4 [cache.host]
5 size_gb = 100.0
6 
7 [cache.disk]
8 size_gb = 200.0
9 use_gds = true
10 storage_path = "/mnt/nvme/kv_cache"
11 
12 [events]
13 enabled = true
14 subject = "kvbm.events"
15 
16 [nixl.backends.UCX]
17 # UCX with default parameters