Production Deployments

This guide covers patterns for running AIPerf benchmarks in production environments — CI/CD pipelines, Kueue-managed clusters, private registries, GitOps workflows, and multi-tenant setups.

CI/CD Integration

Detach Mode

In non-interactive environments (CI pipelines, cron jobs), AIPerf automatically detaches after deploying. You can also force this explicitly:

$
aiperf kube profile \
>
  --config benchmark.yaml \
>
  --image nvcr.io/nvidia/aiperf:latest \
>
  --detach

After deploying, poll for completion:

$
# Wait for the job to complete
$
while true; do
$
  PHASE=$(aiperf kube list my-benchmark 2>/dev/null | awk 'NR==2{print $3}')
$
  [ "$PHASE" = "Completed" ] || [ "$PHASE" = "Failed" ] && break
$
  sleep 30
$
done
$
$
# Download results
$
aiperf kube results my-benchmark --output ./artifacts

aiperf kube results exits nonzero when the job cannot be resolved or the requested download is incomplete, so CI artifact steps fail instead of silently publishing an empty or partial directory.

JSON-Based Monitoring

For automated pipelines, use JSON output:

$
# Preflight check with exit code
$
aiperf kube preflight -o json
$
echo "Exit code: $?"    # 0 = all checks passed, 1 = failures
$
$
# Validation with structured output (takes AIPerfJob/AIPerfSweep CRs, not --config files)
$
aiperf kube validate -o json aiperfjob.yaml

Example GitHub Actions Workflow

1
jobs:
2
  benchmark:
3
    runs-on: self-hosted
4
    steps:
5
      - uses: actions/checkout@v4
6
7
      - name: Preflight check
8
        run: |
9
          aiperf kube preflight \
10
            --endpoint-url http://dynamo-agg-frontend.dynamo-server.svc:8000/v1 \
11
            -o json
12
13
      - name: Run benchmark
14
        run: |
15
          aiperf kube profile \
16
            --config benchmark.yaml \
17
            --image nvcr.io/nvidia/aiperf:${{ github.sha }} \
18
            --name "ci-${{ github.run_number }}" \
19
            --detach
20
21
      - name: Wait for completion
22
        run: |
23
          kubectl wait --for=condition=Complete \
24
            aiperfjob/ci-${{ github.run_number }} --timeout=60m
25
26
      - name: Collect results
27
        if: always()
28
        run: aiperf kube results ci-${{ github.run_number }} --output ./artifacts
29
30
      - uses: actions/upload-artifact@v4
31
        if: always()
32
        with:
33
          name: benchmark-results
34
          path: ./artifacts/

GitOps Workflows

Generate Manifests for Version Control

Instead of deploying from the CLI, generate Kubernetes manifests and commit them to your GitOps repository:

$
# Generate an AIPerfJob CR (operator mode)
$
aiperf kube generate --operator \
>
  --config benchmark.yaml \
>
  --image nvcr.io/nvidia/aiperf:latest \
>
  > deploy/aiperfjob.yaml
$
$
# Or generate raw manifests (no operator needed)
$
aiperf kube generate --no-operator \
>
  --config benchmark.yaml \
>
  --image nvcr.io/nvidia/aiperf:latest \
>
  > deploy/manifests.yaml

Commit the generated YAML and let ArgoCD, Flux, or your GitOps tool apply it.

Operator Mode vs. Raw Manifests

Operator mode also persists the active pod startup blocker in status.startupIssue. This keeps the first-observed timestamp across operator restarts and lets CI distinguish a retryable capacity delay from a stable image, configuration, or placement failure before the terminal Failed condition is set.

For production use, the operator mode is recommended.

Kueue Gang-Scheduling

If your cluster uses Kueue for resource management, AIPerf integrates with it for quota-managed gang-scheduling.

Submit to a Kueue Queue

$
aiperf kube profile \
>
  --config benchmark.yaml \
>
  --image nvcr.io/nvidia/aiperf:latest \
>
  --queue-name gpu-benchmarks \
>
  --priority-class high-priority

Or in YAML:

1
spec:
2
  scheduling:
3
    queueName: gpu-benchmarks
4
    priorityClass: high-priority

When a queue is specified:

1. The JobSet is created in a suspended state
2. Kueue evaluates quota availability
3. Once resources are available, Kueue unsuspends the JobSet
4. The operator detects the transition and monitors normally

aiperf kube list shows Kueue suspension status:

$
aiperf kube list
$
# Shows phase "Queued" while the JobSet awaits Kueue admission

Private Registries

Image Pull Secrets

If your AIPerf image is in a private registry:

$
# Create the pull secret
$
kubectl create secret docker-registry my-registry \
>
  --docker-server=nvcr.io \
>
  --docker-username='$oauthtoken' \
>
  --docker-password='YOUR_TOKEN' \
>
  -n my-benchmarks
$
$
# Reference it when deploying
$
aiperf kube profile ... --image-pull-secrets my-registry

Or in YAML:

1
spec:
2
  podTemplate:
3
    imagePullSecrets:
4
      - {name: my-registry}

API Keys and Secrets

Pass API keys to the benchmark pods without embedding them in the config:

$
# Create a secret with your API key
$
kubectl create secret generic llm-api-key \
>
  --from-literal=api-key='sk-...' \
>
  -n my-benchmarks
$
$
# Reference it as an environment variable. Name the pod variable, and pass
$
# `<secret>/<key>` as its value.
$
aiperf kube profile ... \
>
  --env-from-secrets.OPENAI_API_KEY llm-api-key/api-key

--env-from-secrets is a mapping flag and accepts three equivalent spellings. --annotations, --labels, and --env-vars accept the same three:

$
--env-from-secrets.OPENAI_API_KEY llm-api-key/api-key   # dot-notation
$
--env-from-secrets OPENAI_API_KEY=llm-api-key/api-key   # KEY=VALUE
$
--env-from-secrets '{"OPENAI_API_KEY": "llm-api-key/api-key"}'  # JSON object

Repeat the flag to set more than one entry.

Endpoint credentials are transported out of band from the benchmark ConfigMap. AIPerf writes only redacted placeholders to run_config.json, then each service restores the real values from Secret-backed environment variables at startup. A credentialed endpoint without the matching valueFrom.secretKeyRef mapping is rejected before the JobSet is created.

Do not use --env-vars for these values: literal pod environment values are not Secret-backed and fail the credential transport check.

Or in YAML:

1
spec:
2
  podTemplate:
3
    env:
4
      - name: OPENAI_API_KEY
5
        valueFrom:
6
          secretKeyRef:
7
            name: llm-api-key
8
            key: api-key

Mounting Secret Files

For secrets that need to be files (e.g., certificates, tokens):

$
aiperf kube profile ... \
>
  --secret-mounts '{"name": "tls-cert", "mount_path": "/certs/ca.pem", "sub_path": "ca.pem"}'

--secret-mounts accepts the same shapes --tolerations does: a single JSON object, a JSON array covering several mounts in one token, or the flag repeated once per mount.

Node Placement

Target Specific GPUs

$
aiperf kube profile ... \
>
  --node-selector '{"nvidia.com/gpu.product": "NVIDIA-A100-SXM4-80GB"}'

Tolerate GPU Taints

Many clusters taint GPU nodes. Add tolerations so benchmark pods can schedule there:

$
aiperf kube profile ... \
>
  --tolerations '[{"key": "nvidia.com/gpu", "operator": "Exists", "effect": "NoSchedule"}]'

Or in YAML:

1
spec:
2
  podTemplate:
3
    nodeSelector:
4
      nvidia.com/gpu.product: "NVIDIA-A100-SXM4-80GB"
5
    tolerations:
6
      - key: nvidia.com/gpu
7
        operator: Exists
8
        effect: NoSchedule

Scaling Workers

AIPerf distributes workers across pods automatically. The --total-workers flag sets the total number of workers. The system places 10 workers per pod by default:

A --total-workers value above the per-pod count that is not a multiple of it cannot be expressed as a JobSet of identical pods, so it is rejected with an error naming the two nearest usable totals. Keep the total a multiple of 10, or set benchmark.runtime.workersPerPod to change the packing factor — there is no CLI flag for it. A total at or below the per-pod count runs on a single pod.

Only an authored total is rejected. When you omit --total-workers and benchmark.runtime.workers, the count derived from ceil(concurrency / connectionsPerWorker) is rounded up to a whole number of pods instead, since failing on it would report a worker count you never chose.

Each worker maintains up to connectionsPerWorker concurrent requests (default: 100). When you pass --total-workers explicitly, the CLI derives this from your concurrency and worker count: connectionsPerWorker = ceil(concurrency / workers). Left unset, the relationship inverts and the worker count is derived instead: workers = ceil(concurrency / connectionsPerWorker).

For high-concurrency benchmarks, scale workers rather than increasing connections per worker. This distributes the load across pods and nodes.

Results Server

The operator includes a results server sidecar that provides HTTP access to stored results. This powers aiperf kube results (which retrieves from the operator’s PVC by default) and provides analytics endpoints backed by the SQLite runs index (.aiperf_index.sqlite on the results volume).

Available Endpoints

After the operator is running:

$
# Port-forward to the results server
$
kubectl port-forward -n aiperf-system svc/aiperf-operator 8081:8081
$
$
# List stored results
$
curl localhost:8081/api/v1/results
$
$
# Get summary for a specific job
$
curl localhost:8081/api/v1/analytics/summary/my-benchmarks/my-benchmark
$
$
# Leaderboard across all benchmarks
$
curl localhost:8081/api/v1/analytics/leaderboard
$
$
# Compare two runs (repeat the jobs parameter, one per job)
$
curl "localhost:8081/api/v1/analytics/compare?jobs=run-a&jobs=run-b"
$
$
# Job history
$
curl "localhost:8081/api/v1/analytics/history?model=Qwen/Qwen3-0.6B"

Storage Configuration

Results are stored on the operator’s PVC. Configure retention in the Helm values:

1
operator:
2
  env:
3
    resultsTtlDays: "30"           # auto-cleanup after 30 days
4
    resultsCompressOnDisk: "true"  # zstd compression
5
6
storage:
7
  enabled: true                    # default; backs results in a PVC so they survive pod restarts
8
  size: 1Ti                        # only used when enabled: true
9
  storageClassName: ""             # cluster default; only used when enabled: true

spec.resultsTtlDays overrides this default per AIPerfJob or AIPerfSweep. A sweep persists the selected value in each epoch’s aggregate, so cleanup remains effective after Kubernetes deletes the parent CR. Cleanup also removes the corresponding sweep index rows and repairs the sweep’s latest.txt pointer.

Exposing Results Outside the Cluster (Ingress)

By default the results server is reachable only via ClusterIP + kubectl port-forward. To expose it through an Ingress (e.g. for a shared dashboard link), enable ingress.* in the Helm values:

1
ingress:
2
  enabled: true
3
  className: "nginx"          # empty uses the cluster default IngressClass
4
  annotations: {}
5
  hosts:
6
    - host: aiperf.example.com
7
      paths:
8
        - path: /
9
          pathType: Prefix    # backend port defaults to resultsServer.port (8081)
10
  tls: []                     # optional: list of {hosts: [...], secretName: ...}

The template lives at deploy/helm/aiperf-operator/templates/ingress.yaml and routes to the operator Service on resultsServer.port. Each path may override the backend port via portNumber.

Environment Variables

Fine-tune benchmark behavior with environment variables in the pod template:

1
spec:
2
  podTemplate:
3
    env:
4
      - name: AIPERF_HTTP_CONNECTION_LIMIT
5
        value: "200"
6
      - name: AIPERF_HTTP_KEEPALIVE_TIMEOUT
7
        value: "120"

Probe tunables such as AIPERF_K8S_HEALTH_STARTUP_PERIOD_SECONDS and AIPERF_K8S_HEALTH_STARTUP_FAILURE_THRESHOLD do not belong here. They are read by whichever process renders the JobSet — the operator pod, or the CLI in direct mode — and baked into the pod spec’s probe fields before the benchmark container exists. Setting them under spec.podTemplate.env has no effect; set them on the operator Deployment (or in the CLI’s own environment) instead.

There is no AIPERF_HTTP_TIMEOUT variable; set the per-request timeout with the --request-timeout-seconds CLI flag (or request_timeout_seconds in the benchmark config) instead. See the Environment Variables Reference for the full list.

Multi-Tenant Clusters

Operator Scope and Namespace Ownership

An operator install is one of two things, decided by operator.id:

A scoped operator claims each namespace it watches by writing a coordination.k8s.io/v1 Lease named aiperf-operator into it, and renews that Lease on a timer. Every operator — the global one included — skips any namespace whose Lease is held by a different identity and still fresh. Nothing is set on the AIPerfJob itself, so a job cannot be submitted to the wrong operator by forgetting a label.

This is what lets a scoped build run beside the production operator:

$
helm install aiperf-test deploy/helm/aiperf-operator \
>
  --namespace aiperf-system-test --create-namespace \
>
  --set operator.id=my-branch \
>
  --set 'operator.watchNamespaces={aiperf-test}'

The global operator stops reconciling aiperf-test as soon as the claim lands, and resumes when the claim lapses. aiperf kube list shows the holder in its OWNER column; - means the global operator owns that namespace, and ? means the claim could not be read (usually missing RBAC on leases).

The global operator caches namespace ownership and refreshes it every AIPERF_OPERATOR_CLAIM_LEASE_SECONDS / 3 seconds (~100 s by default). During that window after a scoped operator starts, the global operator may briefly act on events in a newly-claimed namespace before its cache updates.

Two scoped operators cannot claim the same namespace: the second one fails at startup naming the current holder, rather than silently double-reconciling.

The claim’s duration (AIPERF_OPERATOR_CLAIM_LEASE_SECONDS, default 300s) is deliberately long, so a crash-looping operator keeps its namespaces across restarts. Uninstalling a scoped operator lets its claims expire and the namespaces fall back to the global operator; nothing needs to be deleted by hand for ownership to transfer.

The aiperf-operator Lease object itself does survive the uninstall in each watched namespace. It is expired, has no effect on ownership, and is only visible in kubectl get leases. Remove it manually if you want it gone:

$
kubectl delete lease aiperf-operator -n <namespace>

Namespace Isolation

Benchmarks run in the namespace you name with --namespace. For multi-tenant setups, give each team its own:

$
aiperf kube profile ... --namespace team-a-benchmarks --operator
$
aiperf kube profile ... --namespace team-b-benchmarks --operator

By default the operator watches all namespaces for AIPerfJob CRs; set operator.watchNamespaces in the Helm values to restrict it to a fixed list, and operator.id to also claim those namespaces away from a global operator (see Operator Scope and Namespace Ownership). Each job gets its own RBAC scoped to its namespace. Here --operator bypasses cluster-scoped CRD discovery, and the namespaces must already exist — the CLI never issues a Namespace-create request. A tenant therefore needs permission to create AIPerfJob resources in its namespace, not permission to read CRDs or create namespaces.

The per-job Role grants no create, update, or delete verbs, so a compromised benchmark pod cannot launch workloads of its own. What it retains inside its namespace is read access to pod, ConfigMap, Service, Endpoint, Event, and Job metadata, plus patch on its own AIPerfJob and JobSet — which is why a namespace per team, rather than a shared benchmark namespace, is still the tenancy boundary. See RBAC and Security.

If cluster policy forbids chart-managed cluster-scoped RBAC, install with rbac.create=false, serviceAccount.create=false, an explicit serviceAccount.name, and tests.enabled=false; that combination emits zero ClusterRole/ClusterRoleBinding objects. See Eliminating all cluster-scoped RBAC.

Resource Quotas

The preflight checker validates resource quotas before deploying:

$
aiperf kube preflight --namespace team-a-benchmarks --workers 20

If the namespace has a ResourceQuota, preflight projects the total CPU and memory requirements and warns if deployment would exceed the quota.

Operator Management

Upgrading

$
helm upgrade aiperf-operator deploy/helm/aiperf-operator \
>
  --namespace aiperf-system

WARNING: Drain active benchmarks before upgrading from a chart older than 0.8.0. Those versions rendered the benchmark Role and RoleBinding into a chart-owned aiperf-benchmarks namespace; the upgrade deletes that RBAC and recreates it in the release namespace. A benchmark running in aiperf-benchmarks at that moment loses its RBAC mid-flight and starts failing its pod, ConfigMap, and JobSet reads. Wait for in-flight runs to reach a terminal phase (aiperf kube list) before running helm upgrade.

Uninstalling

$
helm uninstall aiperf-operator --namespace aiperf-system

Both CRDs carry helm.sh/resource-policy: keep, so the AIPerfJob and AIPerfSweep CRDs — and any existing custom resources — survive the uninstall. To remove everything:

$
kubectl delete crd aiperfjobs.aiperf.nvidia.com
$
kubectl delete crd aiperfsweeps.aiperf.nvidia.com
$
kubectl delete namespace aiperf-system

Monitoring the Operator

Check operator logs:

$
kubectl logs -n aiperf-system -l app.kubernetes.io/name=aiperf-operator -f

The operator emits structured logs with job events, phase transitions, and error details.

Debugging Failed Jobs

By default the operator cleans up JobSet pods once a job terminates, which makes postmortems hard. Set keepFailedPods: true on the AIPerfJob spec to preserve failed pod attempts for inspection:

1
spec:
2
  keepFailedPods: true   # default: false; retains failed JobSet pods for kubectl logs/exec

This changes JobSet lifecycle (completed pods still get reaped via ttlSecondsAfterFinished); enable it only while diagnosing, then remove for steady-state runs.

Deploying Before the Endpoint Is Reachable

The operator runs a TCP/HTTP reachability probe against spec.benchmark.endpoint before spinning up workers. If the inference server isn’t live yet at deploy time (e.g. GitOps applies the CR before the model pod is Ready), set skipEndpointCheck: true to bypass the probe:

1
spec:
2
  skipEndpointCheck: true   # default: false; skips the operator-side endpoint reachability probe

Workers will still fail fast if the endpoint never comes up — this only suppresses the upfront check.

Related Documentation

• Getting Started — First benchmark walkthrough
• Deploy from a Source Checkout — Build and push AIPerf, Helm install the operator, and run on a real cluster
• Kubernetes Configuration — All CRD fields and deployment options
• Monitoring and Troubleshooting — Watch, debug, and diagnose issues
• Environment Variables — All AIPERF_* environment variables