> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/aiperf/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/aiperf/_mcp/server. # Deploy from a Source Checkout to a Real Kubernetes Cluster This guide starts from an AIPerf source checkout and deploys that code to a real Kubernetes cluster. It builds a container image, pushes it to a registry, installs or upgrades the AIPerf operator with Helm, then runs a benchmark against a real inference endpoint. This guide does **not** use the mock server path. Use it when you have a real OpenAI-compatible endpoint already running, or when you want to deploy a real Dynamo/vLLM endpoint first. ## Prerequisites You need: - An AIPerf source checkout. - `kubectl` configured for the target cluster. - Helm v3. - Docker or another container builder that can build and push OCI images. - Registry credentials for the image repository you will use. - Permission to install CRDs, create the operator namespace, create benchmark namespaces, and create JobSet workloads. - JobSet installed on the cluster. If it is not installed, install it before the AIPerf operator. - A real OpenAI-compatible inference endpoint reachable from benchmark pods, or permission to deploy one. Check the cluster first: ```bash kubectl cluster-info kubectl get nodes kubectl api-resources | grep -i jobset ``` For GPU benchmarks, verify GPU resources are allocatable: ```bash kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu ``` ## 1. Set up the checkout Install the local development environment from the checkout: ```bash make first-time-setup ``` Use the local CLI through `uv run` until the package is installed somewhere else: ```bash uv run aiperf --help uv run aiperf kube --help ``` Pick a tag that identifies the exact checkout you are deploying: ```bash export AIPERF_TAG="$(git rev-parse --short HEAD)" ``` ## 2. Build and push the AIPerf image The same image is used by the operator containers and benchmark JobSet pods unless you explicitly override `defaults.image` in the Helm chart. ### NGC-style registry Use this shape for an NGC organization or private NVIDIA registry namespace: ```bash export AIPERF_IMAGE="nvcr.io//aiperf:${AIPERF_TAG}" docker build -t "${AIPERF_IMAGE}" . docker push "${AIPERF_IMAGE}" ``` ### GitHub Container Registry Use this shape for GHCR: ```bash export AIPERF_IMAGE="ghcr.io//aiperf:${AIPERF_TAG}" docker build -t "${AIPERF_IMAGE}" . docker push "${AIPERF_IMAGE}" ``` If your cluster nodes use a different architecture than the build host, build for the cluster platform: ```bash docker buildx build \ --platform linux/amd64 \ -t "${AIPERF_IMAGE}" \ --push \ . ``` ## 3. Create an image pull secret if the registry is private Skip this section if every node can pull the image without a secret. Create the operator namespace first: ```bash kubectl create namespace aiperf-system --dry-run=client -o yaml | kubectl apply -f - ``` For NGC-style registries: ```bash kubectl create secret docker-registry aiperf-registry \ --namespace aiperf-system \ --docker-server=nvcr.io \ --docker-username='$oauthtoken' \ --docker-password="${NGC_API_KEY}" ``` For GHCR: ```bash kubectl create secret docker-registry aiperf-registry \ --namespace aiperf-system \ --docker-server=ghcr.io \ --docker-username="${GITHUB_USER}" \ --docker-password="${GITHUB_TOKEN}" ``` The Helm install below references this secret for the operator. Benchmark jobs also need pull access in the namespace you run them in. Kubernetes secrets are namespace-scoped, so create the same pull secret there: ```bash kubectl create namespace --dry-run=client -o yaml | kubectl apply -f - kubectl create secret docker-registry aiperf-registry \ --namespace \ --docker-server= \ --docker-username= \ --docker-password= ``` ## 4. Install or upgrade the AIPerf operator Split the image into repository and tag for Helm: ```bash export AIPERF_IMAGE_REPOSITORY="${AIPERF_IMAGE%:*}" export AIPERF_IMAGE_TAG="${AIPERF_IMAGE##*:}" ``` Install the operator: ```bash helm upgrade --install aiperf-operator deploy/helm/aiperf-operator \ --namespace aiperf-system \ --create-namespace \ --set image.repository="${AIPERF_IMAGE_REPOSITORY}" \ --set image.tag="${AIPERF_IMAGE_TAG}" \ --set image.pullPolicy=IfNotPresent ``` If you created `aiperf-registry`, include it in the Helm release: ```bash helm upgrade --install aiperf-operator deploy/helm/aiperf-operator \ --namespace aiperf-system \ --create-namespace \ --set image.repository="${AIPERF_IMAGE_REPOSITORY}" \ --set image.tag="${AIPERF_IMAGE_TAG}" \ --set image.pullPolicy=IfNotPresent \ --set 'imagePullSecrets[0].name=aiperf-registry' ``` The chart default for `defaults.image` is empty, which means AIPerfJob pods use `:`, falling back to the chart's `appVersion` when `image.tag` is also empty. Set `defaults.image` only when benchmark pods should run a different image from the operator. Wait for the operator and results server: ```bash kubectl rollout status deploy/aiperf-operator -n aiperf-system --timeout=180s kubectl get pods -n aiperf-system ``` Run Helm tests if the cluster can pull the chart's test image: ```bash helm test aiperf-operator -n aiperf-system ``` ## 5. Run against an existing real endpoint Use this path when your inference server is already deployed in the cluster or reachable from the cluster network. Set the endpoint and model: ```bash export MODEL="Qwen/Qwen3-0.6B" export ENDPOINT_URL="http://vllm.default.svc.cluster.local:8000/v1" ``` Run cluster-side preflight checks: ```bash uv run aiperf kube preflight \ --image "${AIPERF_IMAGE}" \ --endpoint-url "${ENDPOINT_URL}" \ --workers 8 ``` Submit a benchmark: ```bash uv run aiperf kube profile \ --model "${MODEL}" \ --url "${ENDPOINT_URL}" \ --image "${AIPERF_IMAGE}" \ --total-workers 8 \ --request-count 1000 \ --concurrency 50 \ --streaming ``` For private benchmark images, pass the pull secret name when submitting jobs: ```bash uv run aiperf kube profile \ --model "${MODEL}" \ --url "${ENDPOINT_URL}" \ --image "${AIPERF_IMAGE}" \ --image-pull-secrets aiperf-registry \ --total-workers 8 \ --request-count 1000 \ --concurrency 50 \ --streaming ``` For repeatable runs, put the benchmark configuration in YAML and pass `--config benchmark.yaml`; see [End-to-End Workflow](/aiperf/dev/kubernetes/end-to-end-workflow) for the `init` -> `validate` -> `preflight` -> `profile` sequence. ## 6. Optional: deploy a real Dynamo/vLLM endpoint first Skip this section if you already have a real endpoint. Install the Dynamo platform chart if your cluster does not already have it: ```bash helm upgrade --install dynamo-platform \ oci://nvcr.io/nvidia/ai-dynamo/dynamo-platform \ --version 1.1.0 \ --namespace dynamo-system \ --create-namespace \ --set dynamo-operator.webhook.enabled=false \ --set grove.enabled=false \ --set kai-scheduler.enabled=false ``` Deploy an aggregated Dynamo vLLM server by applying a `DynamoGraphDeployment` manifest such as the aggregated vLLM example in [Getting Started on Kubernetes](/aiperf/dev/kubernetes/getting-started-on-kubernetes#step-2-deploy-a-dynamo-inference-server): ```bash kubectl create namespace dynamo-server --dry-run=client -o yaml | kubectl apply -f - kubectl apply -f dynamo-server.yaml ``` Wait for the frontend and worker pods to become ready: ```bash kubectl get pods -n dynamo-server -w ``` Use the Dynamo service URL as the benchmark endpoint: ```bash export ENDPOINT_URL="http://dynamo-agg-frontend.dynamo-server.svc:8000/v1" uv run aiperf kube preflight \ --image "${AIPERF_IMAGE}" \ --endpoint-url "${ENDPOINT_URL}" \ --workers 8 uv run aiperf kube profile \ --model "${MODEL}" \ --url "${ENDPOINT_URL}" \ --image "${AIPERF_IMAGE}" \ --total-workers 8 \ --request-count 1000 \ --concurrency 50 \ --streaming ``` ## 7. Monitor and retrieve results Watch progress: ```bash uv run aiperf kube list ``` Reattach to a detached run: ```bash uv run aiperf kube attach ``` Download results: ```bash uv run aiperf kube results --output ./aiperf-results ``` Port-forward the results server and dashboard: ```bash kubectl port-forward -n aiperf-system svc/aiperf-operator 8081:8081 ``` Then open `http://localhost:8081`. ## 8. Upgrade after source changes After changing source code, repeat the build and push with a new immutable tag: ```bash export AIPERF_TAG="$(git rev-parse --short HEAD)-$(date +%Y%m%d%H%M%S)" export AIPERF_IMAGE="nvcr.io//aiperf:${AIPERF_TAG}" docker build -t "${AIPERF_IMAGE}" . docker push "${AIPERF_IMAGE}" export AIPERF_IMAGE_REPOSITORY="${AIPERF_IMAGE%:*}" export AIPERF_IMAGE_TAG="${AIPERF_IMAGE##*:}" helm upgrade aiperf-operator deploy/helm/aiperf-operator \ --namespace aiperf-system \ --reuse-values \ --set image.repository="${AIPERF_IMAGE_REPOSITORY}" \ --set image.tag="${AIPERF_IMAGE_TAG}" ``` Wait for the new operator pod before submitting new jobs: ```bash kubectl rollout status deploy/aiperf-operator -n aiperf-system --timeout=180s ``` ## Troubleshooting ### ImagePullBackOff Check the failing pod and events: ```bash kubectl describe pod -n aiperf-system -l app.kubernetes.io/name=aiperf-operator uv run aiperf kube debug ``` Common fixes: - Confirm the image was pushed with the exact tag used by Helm or `aiperf kube profile`. - Create the pull secret in both `aiperf-system` and the benchmark namespace. - Pass `--image-pull-secrets aiperf-registry` to `aiperf kube profile` for private benchmark images. - Use `--set image.pullPolicy=Always` while testing mutable tags; prefer immutable tags for normal use. ### Operator is running but jobs use an old image The operator image and default benchmark image come from the Helm release. Check the rendered CRD default: ```bash kubectl get crd aiperfjobs.aiperf.nvidia.com \ -o jsonpath='{.spec.versions[?(@.name=="v1alpha1")].schema.openAPIV3Schema.properties.spec.properties.image.default}{"\n"}' ``` If you set `defaults.image`, it overrides the benchmark image independently of `image.repository` and `image.tag`. ### Endpoint check fails Verify the endpoint from inside the cluster: ```bash kubectl run curl-check \ --rm -it \ --restart=Never \ --image=curlimages/curl:latest \ -- curl -sf "${ENDPOINT_URL}/models" ``` If the server is still starting and you intentionally want the benchmark to wait until worker runtime, set `skipEndpointCheck: true` in the AIPerfJob YAML. Do not use this to hide a wrong service name or namespace. ### RBAC or namespace errors The Helm chart creates benchmark RBAC in the release namespace and in any namespaces listed in `benchmarkRbacNamespaces`. If you run jobs in a different namespace, add that namespace to `benchmarkRbacNamespaces`: ```bash helm upgrade aiperf-operator deploy/helm/aiperf-operator \ --namespace aiperf-system \ --reuse-values \ --set 'benchmarkRbacNamespaces[0]=team-a-benchmarks' ``` ## Related Documentation - [Getting Started on Kubernetes](/aiperf/dev/kubernetes/getting-started-on-kubernetes) -- First benchmark walkthrough and Dynamo manifest examples. - [End-to-End Workflow](/aiperf/dev/kubernetes/end-to-end-workflow) -- Full `init` -> `validate` -> `preflight` -> `profile` -> `results` lifecycle. - [Production Deployments](/aiperf/dev/kubernetes/production-deployment) -- CI/CD, Kueue, private registries, multi-tenancy, and operations patterns. - [Kubernetes Configuration Reference](/aiperf/dev/kubernetes/kubernetes-configuration-reference) -- CRD fields, Helm values, and AIPerfJob configuration. - [Monitoring and Troubleshooting](/aiperf/dev/kubernetes/monitoring-and-observability) -- Watch, debug, logs, and common failure modes.