Kubernetes Configuration Reference

This guide covers all the ways to configure AIPerf benchmarks on Kubernetes — from the AIPerfJob custom resource fields to CLI flags and Helm chart settings.

AIPerfJob Custom Resource

An AIPerfJob is a Kubernetes custom resource that tells the operator what benchmark to run. Here is the full structure:

1
apiVersion: aiperf.nvidia.com/v1alpha1
2
kind: AIPerfJob
3
metadata:
4
  name: my-benchmark
5
  namespace: my-benchmarks  # optional; kubectl uses your current context's namespace when omitted
6
spec:
7
  # Benchmark configuration (what to measure)
8
  benchmark:
9
    models: ["Qwen/Qwen3-0.6B"]
10
    endpoint:
11
      urls: ["http://dynamo-agg-frontend.dynamo-server.svc:8000/v1"]
12
      streaming: true
13
    datasets:
14
      - name: main
15
        type: synthetic
16
        entries: 1000
17
        prompts:
18
          isl: { mean: 512, stddev: 0 }
19
          osl: { mean: 128, stddev: 0 }
20
    phases:
21
      - name: profiling
22
        type: concurrency
23
        concurrency: 50
24
        requests: 500
25
    artifacts:
26
      autoPlot: true
27
      plotRequired: false
28
29
  # Config-v2 envelope field (a sibling of benchmark, not nested inside it)
30
  plot:
31
    visualization:
32
      single_run_defaults: [ttft_over_time]
33
      single_run_plots:
34
        ttft_over_time:
35
          type: scatter
36
          x: request_number
37
          y: time_to_first_token
38
          title: TTFT over time
39
40
  # Container image (defaults to the chart/AIPerf image when omitted)
41
  image: "nvcr.io/nvidia/aiperf:latest"
42
43
  # Pod resource mode
44
  resourceMode: burstable        # "burstable" (default), "guaranteed", or "none"
45
46
  # Worker scaling
47
  connectionsPerWorker: 100       # max concurrent connections per worker process
48
49
  # Lifecycle
50
  ttlSecondsAfterFinished: 300    # seconds to keep pods after completion
51
  timeoutSeconds: 0               # benchmark timeout (0 = no timeout)
52
53
  # Cancel a running benchmark
54
  cancel: false                   # set to true to cancel
55
56
  # Pod customization
57
  podTemplate:
58
    nodeSelector:
59
      nvidia.com/gpu.product: "A100"
60
    tolerations:
61
      - key: nvidia.com/gpu
62
        operator: Exists
63
        effect: NoSchedule
64
    imagePullSecrets:
65
      - {name: my-registry-secret}
66
    env:
67
      - name: AIPERF_HTTP_CONNECTION_LIMIT
68
        value: "200"
69
    volumes:
70
      - name: model-cache
71
        persistentVolumeClaim:
72
          claimName: model-cache
73
    volumeMounts:
74
      - name: model-cache
75
        mountPath: /root/.cache/huggingface
76
77
  # Kueue scheduling
78
  scheduling:
79
    queueName: my-queue
80
    priorityClass: high-priority

Spec Fields Reference

Benchmark Configuration (spec.benchmark)

The benchmark section mirrors the standard AIPerf YAML config. Any field you use in a local aiperf profile run works here.

See the YAML Config Reference for the complete set of benchmark fields.

Shorthand siblings. The apiserver also accepts the singular shortcuts from AIPerf CLI YAML — model: (string/list/object), dataset: (single dict), and top-level warmup: / profiling: (phase dicts) — and the operator hoists them into the canonical models/datasets/phases shapes before validation. Mixing the canonical and shorthand form for the same slot (e.g. both datasets: and dataset:) is rejected at admission. Full rule catalog: CRD Validation Rules.

datasets and phases are lists, not maps. Kubernetes alphabetizes the keys of object-typed CRD fields at storage time, so phase ordering is only preserved because it is expressed as a list.

Plot envelope (spec.plot)

Kubernetes preserves the Config-v2 plot: envelope field and runs it after the benchmark’s exporters have finished, before the results-sidecar ready marker is written. The resolved envelope is saved as .aiperf-plot-config.yaml beside the run artifacts, so a later aiperf plot <run-directory> uses the same visualization configuration.

Setting plot: implies spec.benchmark.artifacts.autoPlot: true unless autoPlot: false was explicitly authored. With the default plotRequired: false, rendering failures produce a warning and the exported benchmark artifacts still become ready. With plotRequired: true, rendering is part of the completion transaction: a failure leaves results unready and the controller exits non-zero. Inline plot mappings are the portable form for hand-authored CRs; aiperf kube generate -f config.yaml resolves a file-backed plot: ./plots/config.yaml into the inline form when it emits the CR.

Load Phases (spec.benchmark.phases)

Each phase defines a load pattern:

1
phases:
2
  - name: warmup
3
    kind: warmup
4
    type: concurrency
5
    concurrency: 10
6
    requests: 10
7
  - name: profiling
8
    kind: profiling
9
    type: concurrency
10
    concurrency: 50
11
    requests: 500

Deployment Options (spec top level)

spec also carries the remaining Config-v2 envelope fields as siblings of benchmark: plot (below), plus sweep, multiRun, variables, randomSeed, and noSweepTable. sweep and multi-run orchestration (multiRun.numRuns > 1 or multiRun.convergence) are rejected on AIPerfJob and required/allowed on AIPerfSweep — see CRD Validation Rules. AIPerfSweep additionally accepts childMetadata (labels/annotations stamped onto every child AIPerfJob).

Pod Template (spec.podTemplate)

Customize the pods that run your benchmark. Every field is optional; typed fields are preferred over extraPodSpec because preflight checks, env merging, and securityContext merging only apply to typed fields.

Scheduling (spec.scheduling)

For clusters using Kueue for resource management:

CLI Flags

When using aiperf kube profile, you can set deployment options via CLI flags. These override values in a config file:

The four map-valued flags — --annotations, --labels, --env-vars, and --env-from-secrets — each accept three equivalent spellings: --labels tier=gold, --labels.tier gold, and --labels '{"tier": "gold"}'. Repeat the flag for additional entries.

Benchmark CLI flags use the same precedence for plain AIPerf config files and AIPerfJob CR input: explicitly passed flags override YAML, while omitted CLI defaults do not rewrite authored values. This applies to profile and generate; kube sweep applies the same benchmark overrides before it builds the AIPerfSweep template. Kubernetes deployment flags merge into the CR deployment subtree, so, for example, --node-selector gpu=true does not erase an unrelated YAML podTemplate.affinity or podTemplate.volumes block. An explicit list-valued flag still replaces the corresponding YAML list.

Operator mode is auto-detected when neither mode flag is passed. On a multi-tenant cluster where users can create namespaced AIPerfJob resources but cannot read cluster-scoped CRDs, pass --operator explicitly. When an explicit --namespace is also supplied, profile assumes the namespace was pre-provisioned and does not attempt to create it. --operator and --no-operator are mutually exclusive.

When an AIPerfJob CR is passed to profile --no-operator or generate --no-operator, direct mode preserves the fields that JobSet can represent, including imagePullPolicy, resourceMode, keepFailedPods, ttlSecondsAfterFinished, podTemplate, and scheduling. The operator still owns CR lifecycle fields such as timeoutSeconds, resultsTtlDays, cancel, and failurePolicy; they have no direct-mode reconciler.

Some benchmark runtime fields are intentionally Kubernetes-managed: artifacts.dir is fixed to the mounted /results volume, and the operator sets the service run type, API bind, dataset-service URL, UI, and ZMQ transport needed for cross-pod operation. Other runtime fields, including workers, workersPerPod, recordProcessors, recordProcessorsPerPod, and statsInterval, remain user-configurable. A total recordProcessors value must divide evenly across identical worker pods; otherwise set recordProcessorsPerPod explicitly. These fields also drive the preflight memory estimate, so a pod layout you set here is the layout the estimate is sized against — see Memory estimator.

Helm Chart Configuration

The operator Helm chart is configured via values.yaml. Key settings:

Operator

1
operator:
2
  replicas: 1
3
  id: ""                            # "" = cluster-wide operator; a unique id
4
                                    # claims operator.watchNamespaces so the
5
                                    # cluster-wide operator steps aside
6
  resources:
7
    requests: { cpu: 250m, memory: 256Mi }
8
    # No limits set by default (burstable QoS) so the operator can scale
9
    # memory/CPU with high-concurrency runs.
10
  env:
11
    monitorInterval: "10.0"         # seconds between status checks
12
    monitorInitialDelay: "5.0"      # delay before first status check
13
    jobTimeoutSeconds: "0"          # 0 = no timeout
14
    podRestartThreshold: "3"        # restarts before warning events
15
    resultsTtlDays: "30"            # days to keep results on PVC
16
    resultsMaxRetries: "5"          # retries for fetching results
17
    resultsRetryDelay: "2.0"        # delay between result fetch retries
18
    endpointCheckTimeout: "10.0"    # endpoint health check timeout
19
    resultsCompressOnDisk: "true"   # store results as zstd on PVC

operator.replicas is fixed at 1. The kopf process and runs index have one authoritative writer and do not use leader election, so the chart rejects multi-replica values instead of presenting unsafe pseudo-HA.

operator.id decides namespace ownership between installs. Leave it empty for the cluster-wide operator. Set it, together with operator.watchNamespaces, to run a scoped operator that leases those namespaces away from the cluster-wide one — see Operator Scope and Namespace Ownership.

Storage

Results are stored on a PVC so they survive pod deletion:

1
storage:
2
  enabled: true           # default — PVC-backed; set false for ephemeral emptyDir
3
  size: 1Ti
4
  storageClassName: ""    # empty = cluster default
5
  mountPath: "/data"      # mount path inside the operator pod
6
  accessMode: "ReadWriteOnce"
7
  emptyDirSizeLimit: ""   # bounds the fallback emptyDir when enabled=false; empty = unbounded

Results Server

A sidecar that serves stored results via HTTP (used by aiperf kube results by default):

1
resultsServer:
2
  port: 8081
3
  resources:
4
    requests: { cpu: 100m, memory: 512Mi }
5
    limits: { cpu: 500m, memory: 1Gi }

The resultsServer chart block only exposes port and resources today.

The results-server also hosts optional POST routes that create or cancel AIPerfJob resources. These are governed by two environment variables read on the results-server container: AIPERF_OPERATOR_MUTATING_ROUTES_ENABLED (default false) and AIPERF_OPERATOR_MUTATING_ROUTES_TOKEN (default empty — fails closed). When disabled, the read-only APIs stay exposed while those mutating POSTs return 403, so serving the results-server does not grant write access through the operator ServiceAccount. The index-rebuild route is mounted read-only on the results-server, so even with the routes enabled and a valid token it returns 503; restart the operator pod to run the single-writer startup rebuild.

The bundled chart does not template these two variables (there is no resultsServer.mutatingRoutes value and no token-secret projection). To turn the routes on, set both env vars directly on the results-server container — e.g. via a deployment patch or a customized chart template — then have clients send Authorization: Bearer <token> on protected POST requests. The browser dashboard never receives this token and keeps create/cancel controls disabled; use aiperf kube or kubectl from an authenticated terminal for those mutations.

dashboard

Optional Plotly Dash sidecar for the operator Pod. Default off.

See dashboard-ui.md for the full architecture.

Benchmark RBAC Namespaces

1
benchmarkRbacNamespaces: []

The benchmark Role and RoleBinding are installed in the chart’s release namespace whenever rbac.create is true (the default). List additional namespaces in benchmarkRbacNamespaces to install the same pair there — for example when each team runs benchmarks in its own namespace. The chart does not create any of these namespaces; they must already exist.

Operator ServiceAccount

1
serviceAccount:
2
  create: true
3
  name: "" # auto-generated from the release name when create=true
4
  annotations: {}

With create: false the chart provisions no ServiceAccount and no RBAC of its own, so serviceAccount.name is required and must name a pre-provisioned account already bound to the operator’s ClusterRole. Omitting it fails the render; it does not fall back to the namespace default account, which carries none of the operator’s permissions. See rbac-security.md for the out-of-band RBAC tree.

Default Image

The default image used for benchmark jobs if not specified in the CR:

1
defaults:
2
  image: "" # empty = "<image.repository>:<image.tag|Chart.AppVersion>"
3
  imagePullPolicy: "IfNotPresent"

When defaults.image is empty (the chart default), the chart computes the benchmark image as <image.repository>:<image.tag | Chart.AppVersion>, so overriding image.tag automatically propagates to benchmark pods. Set defaults.image explicitly to decouple the benchmark image from the operator image.

Ingress

Expose the results-server HTTP API outside the cluster via a Kubernetes Ingress. Disabled by default — results are reachable via ClusterIP + kubectl port-forward.

1
ingress:
2
  enabled: false
3
  className: ""                    # IngressClass name (e.g. "nginx"); empty uses cluster default
4
  annotations: {}                  # annotations applied to the Ingress
5
  hosts:
6
    - host: aiperf.example.com
7
      paths:
8
        - path: /
9
          pathType: Prefix         # backend port defaults to resultsServer.port; override with portNumber
10
  tls: []                          # optional list of {hosts, secretName}

NetworkPolicy

Restrict pod traffic to/from the operator. Disabled by default — no restrictions applied. When enabled, ingress is allowed from the release namespace, the benchmark namespace, benchmarkRbacNamespaces, and allowedNamespaces on the health (8080), results (resultsServer.port), and metrics (operator.metrics.port, when non-zero) ports. Egress allows DNS, the K8s API server (443/6443), the benchmark namespace, benchmarkRbacNamespaces, and allowedNamespaces.

1
networkPolicy:
2
  enabled: false
3
  allowedNamespaces: []            # extra namespaces allowed to reach the operator and reachable on egress
4
  allowedIngressCIDRs: []          # CIDR allow-list for external scrapers (Prometheus, ingress controllers)

Kueue

1
kueue:
2
  # When set, the benchmark namespace is annotated with
3
  # kueue.x-k8s.io/default-queue-name so all AIPerf jobs are admitted through
4
  # Kueue even without an explicit --queue-name flag. Left empty, it falls
5
  # back to kueue.localQueueName when createQueues is true.
6
  defaultQueueName: ""
7
8
  # Optionally let the chart provision the Kueue objects themselves
9
  # (ResourceFlavor + ClusterQueue + LocalQueue). Off by default so the
10
  # chart renders on clusters without Kueue CRDs.
11
  createQueues: false
12
  flavorName: "default-flavor"
13
  clusterQueueName: "aiperf-cluster-queue"
14
  localQueueName: "aiperf-local-queue"
15
  resources:
16
    cpu: "1000"
17
    memory: "4Ti"
18
    gpu: ""            # empty = omit nvidia.com/gpu from the quota entirely

See Kueue Integration for the full gang-scheduling walkthrough.

helm test hooks

1
tests:
2
  # Renders the two `helm test` hook Pods (CRD check, operator health check)
3
  # and the dedicated ServiceAccount / Role / RoleBinding / ClusterRole /
4
  # ClusterRoleBinding they run under. All five RBAC objects and both Pods are
5
  # gated together, so disabling this never leaves a Pod pointing at a
6
  # ServiceAccount that was not created.
7
  enabled: true

Set tests.enabled=false when cluster policy forbids chart-managed cluster-scoped RBAC. test-crd-installed reads cluster-scoped CustomResourceDefinitions, so its ClusterRole cannot be narrowed to a namespace — this is the only way to make the chart emit zero ClusterRole and ClusterRoleBinding objects. helm test then prints TEST SUITE: None and exits 0. It is a separate flag from rbac.create on purpose; see Eliminating all cluster-scoped RBAC.

Configuration Patterns

Combining CLI Flags with Config Files

CLI flags override config file values. This is useful for changing deployment settings without editing the YAML:

$
# Use config file for benchmark settings, override image and workers
$
aiperf kube profile \
>
  --config benchmark.yaml \
>
  --image my-registry/aiperf:v2.0 \
>
  --total-workers 20 \
>
  --namespace production

Validating Before Deploying

aiperf kube validate checks AIPerfJob and AIPerfSweep CR YAML — files with apiVersion: aiperf.nvidia.com/v1alpha1, a kind:, metadata.name, and spec.benchmark. A plain benchmark config file is not a CR and will fail these structural checks; generate a CR first with aiperf kube generate --operator --config benchmark.yaml.

$
# Validate CR structure and fields
$
aiperf kube validate aiperfjob.yaml
$
$
# Strict mode fails on unknown spec fields
$
aiperf kube validate --strict aiperfjob.yaml
$
$
# JSON output for CI
$
aiperf kube validate -o json aiperfjob.yaml

Preview what will be submitted without deploying:

$
aiperf kube profile --config benchmark.yaml --image aiperf:latest --dry-run

Memory Estimation

AIPerf prints a memory estimate before deploying. This helps you right-size your pods:

$
aiperf kube generate --operator --config benchmark.yaml --image aiperf:latest

The memory estimate is printed to stderr. It accounts for dataset size, number of workers, connection pools, and record buffers.

Multiple Phases

Use multiple phases to warm up before measuring:

1
phases:
2
  - name: warmup
3
    kind: warmup
4
    type: concurrency
5
    concurrency: 10
6
    requests: 20
7
  - name: low_load
8
    kind: profiling
9
    type: concurrency
10
    concurrency: 25
11
    requests: 250
12
  - name: high_load
13
    kind: profiling
14
    type: concurrency
15
    concurrency: 100
16
    requests: 500

Phases run in order. Every phase keeps phase-scoped results; only phases with kind: profiling contribute to profiling aggregates. Canonical names warmup and profiling infer their matching kinds, while custom names require an explicit kind.

Resource Mode

spec.resourceMode controls the QoS class Kubernetes assigns to benchmark pods. The three modes differ only in how requests and limits are emitted onto the manifest — the underlying resource budget is the same in every case.

The mode applies to both controller-pod and worker-pod containers; there is no per-container override. On an AIPerfSweep it also covers the sweep-controller pod (both its sweep-controller and results-sidecar containers), which is why burstable matters there: under sweep.type: adaptive_search that pod imports torch/BoTorch and grows to roughly 350 MiB after the first GP fit, well past its 512Mi request. OOMKill semantics follow the QoS class — guaranteed pods will not be evicted for resource pressure, burstable pods may be throttled, and none/BestEffort pods can be evicted first.

Tunable Environment Variables (AIPERF_K8S_*)

These variables tune the operator and individual benchmark pods. operator.env is a fixed key map, not an arbitrary passthrough: only the keys it declares reach the container. For any other variable, set it on the live Deployment (kubectl -n aiperf-system set env deployment/aiperf-operator -c operator KEY=VALUE — -c operator matters, the Deployment also runs the results-server and dashboard containers) to affect every subsequent job, or use spec.podTemplate.env to affect one CR only.

Resource-sizing and JobSet variables are read by the process that renders the JobSet, so spec.podTemplate.env has no effect on them; they must be set on the operator.

Resource sizing (per-container CPU / memory)

Every control-plane container, the event-bus proxy sidecar, the results sidecar, and the worker pod have a paired _CPU / _MEMORY variable. Defaults are low burstable requests so many tiny jobs can start concurrently; raise them for very large concurrency or high-token workloads.

Architecture toggles

JobSet and lifecycle

Health probes

Ports

All health and service ports are overridable via AIPERF_K8S_PORT_* (e.g. AIPERF_K8S_PORT_API_SERVICE=9090, AIPERF_K8S_PORT_RESULTS_SIDECAR=9091, AIPERF_K8S_PORT_SYSTEM_CONTROLLER_HEALTH=8080). Consult src/aiperf/kubernetes/environment.py::_PortSettings for the full list — changing these is rarely necessary.

The complete, generated reference for every AIPERF_* variable (including non-k8s ones) lives in ../environment-variables.md.

Related Documentation

• Getting Started — First benchmark walkthrough
• Monitoring and Troubleshooting — Live monitoring and debugging
• Production Deployments — CI/CD, Kueue, and GitOps workflows
• CRD Validation Rules — Apiserver-side CEL invariants and shorthand acceptance
• Preflight Checks — What the operator validates before admitting a CR
• Memory Estimator — How per-component memory estimates drive resource requests
• Direct Mode — Trade-offs when running --no-operator
• User-defined output files — artifacts.user_files for templated sidecar files
• YAML Config Reference — Complete benchmark configuration options