Getting Started with AIPerf on Kubernetes

AI Agents (Claude, Copilot, Cursor, etc.): For diagnosing failures, see the AI Agent Debugging Guide.

This guide walks you through benchmarking an NVIDIA Dynamo inference server on Kubernetes using AIPerf. By the end, you will have a cluster running, both operators installed, a benchmark executed against a Dynamo deployment, and your results downloaded.

The same workflow applies to any OpenAI-compatible endpoint (vLLM, TRT-LLM, SGLang) — just change the endpoint URL and model name.

Cluster Setup

If you already have a Kubernetes cluster with GPU nodes and kubectl configured, skip to Prerequisites.

Option A: Use an Existing Cluster

Any Kubernetes v1.24+ cluster with NVIDIA GPUs works. You need:

• kubectl configured to talk to the cluster
• GPU nodes with the NVIDIA device plugin installed
• Permissions to install CRDs and create namespaces

Verify your connection:

$
kubectl cluster-info
$
kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\\.com/gpu

If GPUs show up, skip to Prerequisites.

Option B: Create a Local Kind Cluster with GPU Passthrough

Kind runs Kubernetes inside Docker containers on your local machine. With the NVIDIA container runtime configured as Docker’s default, a Kind node can see the host’s GPUs.

Host requirements:

• Docker with the NVIDIA container runtime as the default runtime
• NVIDIA drivers installed on the host
• kind CLI installed (go install sigs.k8s.io/kind@latest or releases)

One-time Docker setup

If you haven’t already configured Docker for GPU passthrough:

$
# Enable volume-mount-based GPU injection
$
sudo nvidia-ctk config --in-place \
>
  --set accept-nvidia-visible-devices-as-volume-mounts=true
$
$
# Set nvidia as the default Docker runtime
$
sudo nvidia-ctk runtime configure --runtime=docker --set-as-default
$
$
# Restart Docker to pick up changes
$
sudo systemctl restart docker

Verify:

$
docker info 2>/dev/null | grep "Default Runtime"
$
# Should show: Default Runtime: nvidia

Cluster setup

Create the cluster:

$
kind create cluster --name aiperf

Install the NVIDIA device plugin so GPUs become an allocatable resource, and JobSet, which the AIPerf operator uses to run benchmark pods:

$
kubectl --context kind-aiperf apply -f \
>
  https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/master/deployments/static/nvidia-device-plugin.yml
$
$
kubectl --context kind-aiperf apply --server-side -f \
>
  https://github.com/kubernetes-sigs/jobset/releases/latest/download/manifests.yaml

Build the AIPerf image locally and load it into the Kind node so pods can pull it without a registry:

$
docker build -t aiperf:local .
$
kind load docker-image aiperf:local --name aiperf

Verify GPUs are allocatable:

$
kubectl --context kind-aiperf get nodes \
>
  -o jsonpath='{.items[0].status.allocatable.nvidia\.com/gpu}'
$
# Should show: 1 (or more)

Teardown

To delete the cluster when you are done:

$
kind delete cluster --name aiperf

Prerequisites

At this point you should have:

• A Kubernetes cluster with kubectl configured
• GPU nodes with the NVIDIA device plugin
• JobSet installed
• Helm v3 installed locally
• AIPerf installed locally (uv tool install aiperf, or uv sync from a source checkout)
• Access to NGC container registry (nvcr.io/nvidia/ai-dynamo)

Run the preflight checker to verify:

$
aiperf kube preflight

Step 1: Install the Operators

You need two operators: the Dynamo operator (manages inference servers) and the AIPerf operator (manages benchmarks).

Install the Dynamo Operator

$
# Install Dynamo platform (CRDs are bundled into the platform chart in v1.x)
$
helm install dynamo-platform \
>
  oci://nvcr.io/nvidia/ai-dynamo/dynamo-platform \
>
  --version 1.1.0 \
>
  --namespace dynamo-system \
>
  --create-namespace \
>
  --set dynamo-operator.webhook.enabled=false \
>
  --set grove.enabled=false \
>
  --set kai-scheduler.enabled=false

Verify the Dynamo operator is running:

$
kubectl get pods -n dynamo-system

Install the AIPerf Operator

$
helm install aiperf-operator deploy/helm/aiperf-operator \
>
  --namespace aiperf-system \
>
  --create-namespace

For Kind clusters using a locally built image, override the image:

$
helm install aiperf-operator deploy/helm/aiperf-operator \
>
  --namespace aiperf-system \
>
  --create-namespace \
>
  --set image.repository=aiperf \
>
  --set image.tag=local \
>
  --set image.pullPolicy=Never

Verify it is running:

$
kubectl get pods -n aiperf-system

You should see 2/2 containers ready by default (operator + results-server). With the optional Plotly dashboard enabled (dashboard.enabled=true), the count becomes 3/3 (operator + results-server + dashboard).

Step 2: Deploy a Dynamo Inference Server

Create a DynamoGraphDeployment. This example deploys Qwen3-0.6B in aggregated mode using the vLLM backend:

1
# dynamo-server.yaml
2
apiVersion: nvidia.com/v1alpha1
3
kind: DynamoGraphDeployment
4
metadata:
5
  name: dynamo-agg
6
  namespace: dynamo-server
7
spec:
8
  services:
9
    Frontend:
10
      dynamoNamespace: dynamo-agg
11
      componentType: frontend
12
      replicas: 1
13
      extraPodSpec:
14
        mainContainer:
15
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.1.0
16
          env:
17
            - name: POD_UID
18
              valueFrom:
19
                fieldRef:
20
                  fieldPath: metadata.uid
21
    VllmWorker:
22
      dynamoNamespace: dynamo-agg
23
      componentType: worker
24
      replicas: 1
25
      resources:
26
        limits:
27
          gpu: "1"
28
      extraPodSpec:
29
        runtimeClassName: nvidia
30
        mainContainer:
31
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.1.0
32
          workingDir: /workspace/examples/backends/vllm
33
          command: ["python3", "-m", "dynamo.vllm"]
34
          args:
35
            - "--model"
36
            - "Qwen/Qwen3-0.6B"
37
          env:
38
            - name: POD_UID
39
              valueFrom:
40
                fieldRef:
41
                  fieldPath: metadata.uid
42
            - name: DYN_SYSTEM_PORT
43
              value: "9090"

Apply it:

$
kubectl create namespace dynamo-server
$
kubectl apply -f dynamo-server.yaml

Wait for the server to be ready (model loading can take a few minutes):

$
# Watch pods until all are Running
$
kubectl get pods -n dynamo-server -w

Verify the endpoint is healthy:

$
kubectl run curl-test --rm -it --restart=Never --image=curlimages/curl -- \
>
  curl -s http://dynamo-agg-frontend.dynamo-server.svc:8000/v1/models

Step 3: Run Your First Benchmark

Now benchmark the Dynamo server. The Dynamo endpoint URL follows the pattern http://{deployment-name}-frontend.{namespace}.svc:8000/v1:

$
aiperf kube profile \
>
  --model Qwen/Qwen3-0.6B \
>
  --url http://dynamo-agg-frontend.dynamo-server.svc:8000/v1 \
>
  --image nvcr.io/nvidia/aiperf:latest \
>
  --total-workers 10 \
>
  --request-count 500 \
>
  --concurrency 50 \
>
  --streaming

On a Kind cluster with a locally built image, point --image at the loaded tag and disable pulling. Substitute the Service URL of whatever OpenAI-compatible endpoint you are testing against:

$
aiperf kube profile \
>
  --model Qwen/Qwen3-0.6B \
>
  --url http://my-endpoint.default.svc:8000 \
>
  --image aiperf:local \
>
  --image-pull-policy Never \
>
  --total-workers 10 \
>
  --request-count 200 \
>
  --concurrency 5 \
>
  --streaming

What happens:

1. AIPerf builds an AIPerfJob custom resource from your flags
2. Submits it to the cluster, where the AIPerf operator picks it up
3. The operator creates RBAC, a ConfigMap, and a JobSet with a controller pod and worker pods
4. Workers send requests to the Dynamo frontend
5. AIPerf polls the AIPerfJob status and streams progress to your terminal

You will see the job phase, worker readiness (ready/total), and each status condition as the operator reports it. In --no-operator mode AIPerf attaches to the controller pod and tails its output instead.

Press Ctrl+C to detach. The benchmark continues running in the cluster. To cancel it, run aiperf kube cancel or patch the CR directly:

$
kubectl patch aiperfjob <name> -n my-benchmarks --type=merge -p '{"spec":{"cancel":true}}'

Step 4: Using a Config File

For repeatable benchmarks, use an AIPerfJob YAML file. Generate a starter template:

$
aiperf kube init --output benchmark.yaml

Edit it for your Dynamo deployment:

1
apiVersion: aiperf.nvidia.com/v1alpha1
2
kind: AIPerfJob
3
metadata:
4
  name: dynamo-benchmark
5
spec:
6
  benchmark:
7
    models:
8
      - "Qwen/Qwen3-0.6B"
9
    endpoint:
10
      urls:
11
        - "http://dynamo-agg-frontend.dynamo-server.svc:8000/v1"
12
      streaming: true
13
    datasets:
14
      - name: main
15
        type: synthetic
16
        entries: 1000
17
        prompts:
18
          isl:
19
            mean: 512
20
            stddev: 0
21
          osl:
22
            mean: 128
23
            stddev: 0
24
    phases:
25
      - name: warmup
26
        kind: warmup
27
        type: concurrency
28
        concurrency: 10
29
        requests: 20
30
      - name: profiling
31
        kind: profiling
32
        type: concurrency
33
        concurrency: 50
34
        requests: 500

Validate the config before deploying:

$
aiperf kube validate benchmark.yaml

Run it:

$
aiperf kube profile --config benchmark.yaml --image nvcr.io/nvidia/aiperf:latest

Or apply it directly with kubectl (the operator picks it up automatically):

$
kubectl apply -f benchmark.yaml

Step 5: Disaggregated Inference

Dynamo’s disaggregated mode separates prefill and decode into different pods for better GPU utilization. To benchmark a disaggregated deployment:

1. Deploy Dynamo in disaggregated mode (separate prefill and decode workers with KV cache transfer):

1
# dynamo-disagg.yaml
2
apiVersion: nvidia.com/v1alpha1
3
kind: DynamoGraphDeployment
4
metadata:
5
  name: dynamo-disagg
6
  namespace: dynamo-server
7
spec:
8
  services:
9
    Frontend:
10
      dynamoNamespace: dynamo-disagg
11
      componentType: frontend
12
      replicas: 1
13
      extraPodSpec:
14
        mainContainer:
15
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.1.0
16
          env:
17
            - name: POD_UID
18
              valueFrom:
19
                fieldRef:
20
                  fieldPath: metadata.uid
21
            - name: DYN_ROUTER_MODE
22
              value: "kv"
23
    VllmPrefillWorker:
24
      dynamoNamespace: dynamo-disagg
25
      componentType: worker
26
      subComponentType: prefill
27
      replicas: 1
28
      resources:
29
        limits:
30
          gpu: "1"
31
      extraPodSpec:
32
        runtimeClassName: nvidia
33
        mainContainer:
34
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.1.0
35
          workingDir: /workspace/examples/backends/vllm
36
          command: ["python3", "-m", "dynamo.vllm"]
37
          args:
38
            - "--model"
39
            - "Qwen/Qwen3-0.6B"
40
            - "--is-prefill-worker"
41
            - "--connector"
42
            - "kvbm"
43
          env:
44
            - name: POD_UID
45
              valueFrom:
46
                fieldRef:
47
                  fieldPath: metadata.uid
48
            - name: DYN_KVBM_CPU_CACHE_GB
49
              value: "1"
50
    VllmDecodeWorker:
51
      dynamoNamespace: dynamo-disagg
52
      componentType: worker
53
      subComponentType: decode
54
      replicas: 1
55
      resources:
56
        limits:
57
          gpu: "1"
58
      extraPodSpec:
59
        runtimeClassName: nvidia
60
        mainContainer:
61
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.1.0
62
          workingDir: /workspace/examples/backends/vllm
63
          command: ["python3", "-m", "dynamo.vllm"]
64
          args:
65
            - "--model"
66
            - "Qwen/Qwen3-0.6B"
67
            - "--is-decode-worker"
68
          env:
69
            - name: POD_UID
70
              valueFrom:
71
                fieldRef:
72
                  fieldPath: metadata.uid
73
            - name: DYN_SYSTEM_PORT
74
              value: "9090"

2. Benchmark it — the endpoint URL changes to match the deployment name:

$
aiperf kube profile \
>
  --model Qwen/Qwen3-0.6B \
>
  --url http://dynamo-disagg-frontend.dynamo-server.svc:8000/v1 \
>
  --image nvcr.io/nvidia/aiperf:latest \
>
  --total-workers 20 \
>
  --request-count 1000 \
>
  --concurrency 100 \
>
  --streaming

Step 6: View Results

After the benchmark completes, retrieve your results:

$
aiperf kube results

This downloads the full results package including:

• profile_export_aiperf.json — Summary metrics (throughput, latency percentiles, TTFT, ITL)
• inputs.json — Dataset that was used
• server_metrics_export.json — Dynamo server metrics (frontend throughput, KV cache stats, component latencies); empty when discovery found no scrapable endpoints

To retrieve results from a specific job:

$
aiperf kube results dynamo-benchmark

Results are stored on the operator’s persistent volume by default, so aiperf kube results works even after benchmark pods are deleted. To retrieve directly from the benchmark pods instead (downloads every artifact through the controller’s results API, so the controller pod must still be running):

$
aiperf kube results dynamo-benchmark --from-pods

Adding --summary-only narrows the download to the summary files and falls back to kubectl cp when the controller API is unreachable.

Dynamo Server Metrics

When benchmarking Dynamo, AIPerf discovers and collects Prometheus metrics from pods with the nvidia.com/metrics-enabled=true label (or a recognizable inference-server image). By default discovery only searches the benchmark job’s own namespace — the namespace the chart-provisioned benchmark RBAC can list pods in. If Dynamo runs in a different namespace (e.g. dynamo-server), set server_metrics.discovery.namespace: dynamo-server in the benchmark config and grant pod-read access there by adding that namespace to the chart’s serverMetricsDiscoveryNamespaces value — a plain string entry binds the benchmark namespaces’ default ServiceAccount, so benchmark pods running under a custom podTemplate.serviceAccountName need the {namespace, serviceAccounts} entry form (or a manual RoleBinding); see the server-metrics guide for the full RBAC prerequisite table and how to tune or disable discovery. Discovered metrics include:

• Frontend metrics — dynamo_frontend_requests, dynamo_frontend_time_to_first_token_seconds, dynamo_frontend_inter_token_latency_seconds, dynamo_frontend_output_tokens
• Component metrics — Per-worker dynamo_component_requests, dynamo_component_kvstats_gpu_cache_usage_percent, dynamo_component_kvstats_gpu_prefix_cache_hit_rate

These are exported alongside the standard AIPerf metrics in the results package.

Step 7: List and Manage Jobs

See all benchmark jobs across namespaces:

$
aiperf kube list

NAME                 NAMESPACE           OWNER  PHASE      WORKERS  PROGRESS  THROUGHPUT  LATENCY   AGE
dynamo-benchmark     my-benchmarks       -      Completed  10/10    100%      142.3 rps   187.0 ms  5m
disagg-test          my-benchmarks       -      Running    10/10    42%       98.1 rps    201.4 ms  2m

WORKERS is ready/total and LATENCY is the p99 request latency. OWNER is the scoped operator holding that namespace’s claim, - when the cluster-wide operator reconciles it, or ? when the claim could not be read. Use --wide to add model, endpoint, and error columns.

Filter by status:

$
aiperf kube list --running
$
aiperf kube list --completed
$
aiperf kube list --failed

Watch jobs with live refresh:

$
aiperf kube list --watch

Web Dashboard

The AIPerf operator includes a built-in web dashboard for monitoring benchmarks and analyzing results.

Access it by port-forwarding to the operator:

$
kubectl port-forward -n aiperf-system deploy/aiperf-operator 8081:8081

Then open http://localhost:8081 in your browser.

The dashboard provides:

• Dashboard — Overview with KPI cards, active jobs, and throughput trends
• Jobs — Sortable table of all benchmark jobs with phase filters
• Job Detail — Live metrics, charts, phase progress, and pod status for a single job
• Leaderboard — Rank benchmark runs by any metric
• Compare — Side-by-side comparison of multiple jobs
• History — Time-series charts showing metrics across runs
• Sweeps — AIPerfSweep listings and per-sweep variation drill-down

Use Ctrl+K to quickly search and navigate between jobs and pages.

Dynamo Deployment Modes

Backends

Dynamo supports three inference backends. Change the worker image and command:

Without the Operator

If you cannot install the AIPerf operator (e.g., limited cluster permissions), AIPerf can deploy benchmarks directly using raw Kubernetes manifests. Use --no-operator:

$
aiperf kube profile \
>
  --model Qwen/Qwen3-0.6B \
>
  --url http://dynamo-agg-frontend.dynamo-server.svc:8000/v1 \
>
  --image nvcr.io/nvidia/aiperf:latest \
>
  --no-operator

This creates RBAC, ConfigMap, and JobSet directly in an existing namespace. You lose operator features (automated monitoring, results storage, conditions) but the benchmark itself works the same way.

To generate the manifests without applying them (useful for GitOps):

$
aiperf kube generate --no-operator \
>
  --model Qwen/Qwen3-0.6B \
>
  --url http://dynamo-agg-frontend.dynamo-server.svc:8000/v1 \
>
  --image nvcr.io/nvidia/aiperf:latest \
>
  > manifests.yaml

Quick Reference

Next Steps

• Deploy from a Source Checkout — Build and push AIPerf, install the operator with Helm, and run on a real cluster
• Kubernetes Configuration Reference — All CRD fields, deployment options, and config patterns
• Monitoring and Troubleshooting — Watch, debug, and diagnose benchmark issues
• Production Deployments — CI/CD, Kueue scheduling, secrets, and GitOps workflows

Related Documentation

• YAML Config Reference — General AIPerf YAML configuration
• Sequence Length Distributions — ISL/OSL distribution configuration
• Architecture — AIPerf internal architecture