This page is for version dev.
For other versions, use one of these documentation indexes:
- 1.8.1 (default): https://docs.nvidia.com/networking/cloudai/1.8.1/llms.txt
- 1.7.1: https://docs.nvidia.com/networking/cloudai/1.7.1/llms.txt
- 1.6.1: https://docs.nvidia.com/networking/cloudai/1.6.1/llms.txt
- dev: https://docs.nvidia.com/networking/cloudai/dev/llms.txt
This workload (test_template_name is NIXLKVBench) runs NIXL KV-cache benchmarking for key-value store performance testing.
Usage Examples
Test TOML example:
name = "my_nixl_kvbench_test"description = "Example NIXL KVBench test"test_template_name = "NIXLKVBench"[cmd_args]docker_image_url = "<docker container url here>"model = "./examples/model_deepseek_r1.yaml"model_config = "./examples/block-tp1-pp16.yaml"backend = "POSIX"num_requests = 1source = "file"num_iter = 16page_size = 256filepath = "/data"
Test Scenario example:
name = "nixl-kvbench-test"[[Tests]]id = "kvbench.1"num_nodes = 1time_limit = "00:10:00"test_name = "my_nixl_kvbench_test"
Test-in-Scenario example:
name = "nixl-kvbench-test"[[Tests]]id = "kvbench.1"num_nodes = 1time_limit = "00:10:00"name = "my_nixl_kvbench_test"description = "Example NIXL KVBench test"test_template_name = "NIXLKVBench"[Tests.cmd_args]docker_image_url = "<docker container url here>"backend = "UCX"source = "memory"op_type = "READ"
API Documentation
Command Arguments
NIXLKVBenchCmdArgs
Command line arguments for NIXL KVBench.
| Field | Type | Default | Description |
|---|---|---|---|
filepath | str | None | None | Directory path (in container) for storage operations. Example: /data |
total_buffer_size | str | list[str] | None | None | Total buffer size in bytes. Examples: 1024, 1kb, 1mb, 1gb. Use with device_list. The size will be passed into NIXL as integer (bytes) |
device_list | str | list[str] | None | None | Device specs in format ‘id:type:path’ (e.g., ‘11:F:/store0.bin,27:K:/dev/nvme0n1’) |
docker_image_url | str | Required | URL of the Docker image to use for the benchmark. |
etcd_path | str | 'etcd' | Path to the etcd executable. |
wait_etcd_for | int | 60 | Number of seconds to wait for etcd to become healthy. |
etcd_image_url | str | None | None | Optional URL of the Docker image to use for etcd, by default etcd will be run from the same image as the benchmark. |
command | Literal['profile'] | 'profile' | |
kvbench_script | str | '/workspace/nixl/benchmark/kvbench/main.py' | |
python_executable | str | 'python' | |
model_cfg | str | list[str] | None | None | Path to model configuration file used by NIXL KVBench. |
backend | str | list[str] | None | None |
Test Definition
NIXLKVBenchTestDefinition
Test definition for NIXL KVBench.
| Field | Type | Default | Description |
|---|---|---|---|
name | str | Required | |
description | str | Required | |
test_template_name | str | Required | |
cmd_args | cloudai.workloads.nixl_kvbench.nixl_kvbench.NIXLKVBenchCmdArgs | Required | |
dse_excluded_args | list[str] | Factory: list | |
extra_env_vars | dict[str, str | typing.List[str]] | {} | |
extra_cmd_args | dict[str, str] | {} | |
extra_container_mounts | list[str] | [] | |
git_repos | list[cloudai._core.installables.git_repo.GitRepo] | [] | |
nsys | cloudai.models.workload.NsysConfiguration | None | None | |
predictor | cloudai.models.workload.PredictorConfig | None | None | |
training_report | cloudai.models.workload.TrainingReportConfig | None | None | |
agent | str | 'grid_search' | |
agent_steps | int | 1 | |
agent_metrics | list[str] | ['default'] | |
agent_reward_function | str | 'inverse' | |
agent_config | dict[str, Any] | None | None | Agent configuration. |
env_params | dict[str, cloudai.configurator.env_params.EnvParamSpec] | Factory: dict | Environment parameters sampled by the env per trial. Sibling to cmd_args; not part of the agent’s action space. CloudAIGymEnv samples, persists them in the trajectory output, and includes them in the trajectory cache key. |