> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/aiperf/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/aiperf/_mcp/server. # Monitoring and Troubleshooting AIPerf provides several tools for monitoring running benchmarks, diagnosing problems, and retrieving logs. This guide covers how to use each one. --- ## Attaching to a Benchmark The `attach` command connects to a running benchmark via port-forward and streams live progress updates via WebSocket: ```bash aiperf kube attach ``` `aiperf kube profile` follows the run automatically after deploying unless you pass `--detach`, but in operator mode (the default) it polls the AIPerfJob CR rather than port-forwarding. Use `attach` to get the WebSocket progress stream, to reconnect after detaching, or to watch from a different terminal. Attach to a specific job: ```bash aiperf kube attach my-benchmark ``` Press **Ctrl+C** to detach. This only closes your local stream; the benchmark keeps running in the cluster. To stop the benchmark, run `aiperf kube cancel `, which patches `spec.cancel: true` so the operator tears down the JobSet and stamps `status.phase=Cancelled`. --- ## Listing Jobs See all benchmark jobs and their status: ```bash aiperf kube list ``` ``` NAME NAMESPACE OWNER PHASE WORKERS PROGRESS THROUGHPUT LATENCY AGE qwen3-benchmark my-benchmarks - Running 10/10 67% 142.3 rps 318.7 ms 3m llama-throughput my-benchmarks - Completed 10/10 100% 137.9 rps 330.1 ms 15m mistral-test my-benchmarks - Failed 0/4 - - - 20m ``` `WORKERS` is `ready/total`, `THROUGHPUT` is requests per second, and `LATENCY` is the p99 request latency. A dash means the CR has not published that value. `OWNER` names the scoped operator holding that namespace's claim; `-` means the cluster-wide operator reconciles it and `?` means the claim was unreadable. ### Filter by Status ```bash aiperf kube list --running aiperf kube list --completed aiperf kube list --failed ``` ### Wide Output Show additional columns like model name, endpoint, and error messages: ```bash aiperf kube list --wide ``` ### Live Refresh Watch the list with automatic updates: ```bash aiperf kube list --watch aiperf kube list --watch --interval 10 ``` --- ## Reading Logs Get logs from all pods associated with a benchmark: ```bash aiperf kube logs ``` ### Specific Job ```bash aiperf kube logs my-benchmark ``` ### Specific Container The controller pod runs `control-plane` (the SystemController) alongside per-service containers (`dataset-manager`, `timing-manager`, `records-manager`, `api`, `event-bus-proxy`, `results-sidecar`, and optionally `gpu-telemetry-manager` / `server-metrics-manager`). Worker pods run `worker-group-manager`. To see logs from a specific container: ```bash aiperf kube logs --container control-plane ``` ### Follow Logs in Real-Time ```bash aiperf kube logs -f ``` ### Last N Lines ```bash aiperf kube logs --tail 100 ``` ### Save Logs to Files Save all pod logs to a directory (one file per pod): ```bash aiperf kube logs --output ./my-logs ``` The closing line counts what actually reached disk (`Saved logs for 7 of 8 pod(s) to ./my-logs/logs/`), and any pod whose `kubectl logs` call failed gets its own warning with the exit code and stderr. When nothing was written the closing line is a warning, not a success. ### Exit Codes in Scripts `attach` and `logs` exit `1` when the benchmark you named does not exist, so they work as CI existence checks. A benchmark that exists but has no pods left (TTL-collected) still exits `0`. Pass `--ignore-not-found` — same spelling and meaning as `kubectl`'s — to force exit `0` for a missing benchmark in teardown scripts. The other addressing commands (`cancel`, `delete`, `shutdown`, `debug`, `list`) always exit `0`; see the [exit-code convention](/aiperf/dev/kubernetes/end-to-end-workflow#exit-code-convention-for-target-addressing-commands). --- ## Debugging Failed Benchmarks The `debug` command runs a one-shot diagnostic analysis of your deployment: ```bash aiperf kube debug -n my-benchmarks ``` It inspects: - **Pod states** -- Identifies CrashLoopBackOff, ImagePullBackOff, ErrImagePull, OOMKilled, CreateContainerConfigError, RunContainerError, and Unschedulable, each with a suggested fix - **Kubernetes events** -- Shows the most recent warning events (or all recent events with `--verbose`) - **Node resources** -- Reports CPU, memory, and GPU allocatable vs. capacity for each node - **Benchmark diagnostics** -- When `--job-id` targets a specific job, runs the metric detectors in `aiperf.kubernetes.benchmark_diagnosis` over `status.liveMetrics`: high error rate, high tail latency (p99 above a multiple of the average), and a stalled job (Pending too long, or Running with neither throughput nor completed requests). Thresholds are the `AIPERF_K8S_DIAGNOSIS_*` environment variables. The section is omitted when nothing tripped. - **Container logs** -- With `--verbose`, fetches recent logs from problem pods ### Debug a Specific Job ```bash aiperf kube debug --job-id my-benchmark ``` ### Verbose Mode Includes container logs from pods with problems: ```bash aiperf kube debug --job-id my-benchmark --verbose ``` ### All Namespaces Scan every namespace that has AIPerf deployments: ```bash aiperf kube debug --all-namespaces ``` ### Sample Output ``` Diagnostic Report: my-benchmarks POD STATUS RESTARTS NODE ISSUES aiperf-bench-controller-0-0 Running 0 node-1 0 aiperf-bench-workers-0-0 Running 3 node-2 1 Problems Found [aiperf-bench-workers-0-0] OOMKilled (previous) (container: worker-0) Suggestion: Container was killed due to out-of-memory. Increase memory limits. Node Resources NODE READY CPU MEMORY GPU PRESSURE node-1 Yes 8/16 32Gi/64Gi 2/4 - node-2 Yes 4/8 8Gi/16Gi 1/2 MemoryPressure Summary Pods: 2 total, 2 running, 1 with issues Warning events: 3 Nodes under pressure: node-2 ``` --- ## Pre-Flight Checks Before deploying, validate that the cluster is ready: ```bash aiperf kube preflight ``` It runs these checks in order, and stops early only if cluster connectivity fails: - Cluster connectivity and Kubernetes version - Namespace exists (AIPerf never creates it) - RBAC permissions in the target namespace - JobSet CRD installed and JobSet controller running - Resource quotas and node resources - Referenced secrets and image-pull access - Network policies and DNS resolution - Endpoint connectivity (when `--endpoint-url` is given) ### With Specific Parameters ```bash aiperf kube preflight \ --image nvcr.io/nvidia/aiperf:latest \ --endpoint-url http://my-server:8000 \ --workers 20 \ --namespace my-benchmarks ``` ### JSON Output For CI/CD pipelines: ```bash aiperf kube preflight -o json ``` Returns a structured JSON object with pass/fail/warn status for each check, suitable for automated gating. --- ## Retrieving Results After a benchmark completes, get the results: ```bash aiperf kube results ``` By default this downloads the full results package from the operator's PVC storage (use `--from-pods` to pull from the benchmark pods instead). The results include: - `profile_export_aiperf.json` -- Summary metrics - `profile_export.jsonl` -- Per-request timing data - Server metrics and other exported files ### From the Operator Storage Even after pods are deleted, results are stored on the operator's PVC. This is the default: ```bash aiperf kube results my-benchmark ``` ### Summary Only Download only the summary results (faster): ```bash aiperf kube results --summary-only ``` ### Direct From Pods If you want to fetch results directly from the running benchmark pods (via the controller API): ```bash aiperf kube results --from-pods ``` The default `--all` path uses the controller API and nothing else — if that call fails, the download fails. The `kubectl cp` fallback applies only to `--summary-only`, which tries the API first and then copies from the `control-plane` container. ### Shut Down After Download Free up cluster resources by shutting down the API service after downloading (only takes effect with `--from-pods`): ```bash aiperf kube results --from-pods --shutdown ``` ### Save to Custom Directory ```bash aiperf kube results --output ./my-results ``` --- ## Common Issues and Solutions ### Expired Local Kubernetes Login When an interactive `aiperf kube` command receives HTTP 401 or a recognized `kubectl` logged-out response, it pauses and reloads the selected kubeconfig until your normal credential provider works again. Complete the usual login (for example, your OIDC, cloud, or access-proxy login) in another terminal. AIPerf does not launch the provider or request credentials itself. Press **Ctrl+C** to stop waiting. This behavior is limited to terminals with interactive input and output. Operator pods, in-cluster service accounts, redirected output, and CI fail immediately so unattended work cannot hang. HTTP 403 is also returned immediately because it means the authenticated identity lacks RBAC permission, not that the login expired. Missing credential-provider executables, malformed plugins, TLS failures, and unreachable API servers remain ordinary errors. ### Pods Stuck in Pending **Symptom:** `aiperf kube list` shows `Pending` and pods never start. **Diagnosis:** ```bash aiperf kube debug --job-id my-benchmark ``` **Common causes:** - Insufficient resources (CPU, memory, GPU) -- check node capacity in the debug output - Missing node selectors or tolerations -- pods may be targeting nodes that don't exist - Kueue quota exhausted -- check your ClusterQueue capacity In operator mode, inspect the durable startup diagnosis directly: ```bash kubectl get aiperfjob my-benchmark \ -o jsonpath='{.status.startupIssue}{"\n"}{.status.conditions[?(@.type=="WorkersReady")]}{"\n"}' ``` Temporary capacity shortages, pending PVC binding, Kueue admission, and unknown scheduler reasons remain retryable even when `timeoutSeconds: 0`. Stable image, container-configuration, crash-loop, node-selector, untolerated-taint, and volume-affinity blockers fail only after the critical startup grace period. Tune warning and critical thresholds with `AIPERF_K8S_WATCHDOG_PENDING_THRESHOLD_SECONDS` and `AIPERF_K8S_WATCHDOG_PENDING_CRITICAL_THRESHOLD_SECONDS` on the operator container. ### ImagePullBackOff **Symptom:** Pods fail with `ImagePullBackOff`. **Fix:** Verify the image exists and pull secrets are configured: ```bash # Check preflight aiperf kube preflight --image your-image:tag # Add pull secrets aiperf kube profile ... --image-pull-secrets my-registry-secret ``` ### OOMKilled **Symptom:** Worker pods restart with `OOMKilled` status. **Fix:** Reduce concurrency per worker pod so each pod uses less memory: ```yaml spec: connectionsPerWorker: 50 # reduce from default 100 ``` `spec.connectionsPerWorker` is immutable after creation, so changing it means creating a new AIPerfJob. The per-pod memory budget is resolved by the process that renders the JobSet, so it is set on the operator container rather than in the CR. The chart has no values key for it; patch the Deployment directly: ```bash kubectl set env -n aiperf-system deploy/aiperf-operator \ AIPERF_K8S_WORKER_POD_MEMORY=8Gi ``` Per job, `spec.resourceMode` selects the QoS shape: `burstable` (the default) sets requests without limits so containers are not cgroup-OOM-killed for exceeding their request, `guaranteed` sets `requests == limits`, and `none` omits both. It is also immutable after creation. ### Benchmark Timeout **Symptom:** Job transitions to `Failed` with timeout error. **Fix:** Increase or disable the timeout: ```yaml spec: timeoutSeconds: 3600 # 1 hour # or timeoutSeconds: 0 # no timeout ``` ### Endpoint Unreachable **Symptom:** Operator reports endpoint health check failure. **Diagnosis:** Verify the Dynamo frontend is reachable from inside the cluster: ```bash kubectl run curl-test --rm -it --image=curlimages/curl -- \ curl -s http://dynamo-agg-frontend.dynamo-server.svc:8000/v1/models ``` **Fix:** Ensure the Dynamo deployment is healthy (`kubectl get pods -n dynamo-server`), and the URL in your config uses the correct frontend service DNS name: `http://{deploy-name}-frontend.{namespace}.svc:8000/v1`. ### Stale Namespaces The CLI-side watchdog that runs while `aiperf kube profile` follows a benchmark warns when it finds more than two leftover `aiperf-*` namespaces. Remove the finished benchmarks inside one with `aiperf kube cleanup`, or delete the namespace outright: ```bash aiperf kube cleanup --namespace aiperf-bench-old --force kubectl delete namespace aiperf-bench-old ``` --- ## Web Dashboard The AIPerf operator includes a built-in web dashboard for comprehensive monitoring and analysis of your benchmarks. Access it by port-forwarding to the operator: ```bash kubectl port-forward -n aiperf-system deploy/aiperf-operator 8081:8081 ``` Then open [http://localhost:8081](http://localhost:8081) in your browser. ### Dashboard Features - **Dashboard Tab** -- Overview with KPI cards, active jobs count, and throughput trends across all jobs - **Jobs Tab** -- Sortable table of all benchmark jobs with phase filters (Running, Completed, Failed) - **Job Detail Page** -- Live metrics, charts, phase progress bar, and pod status for a single job - **Sweeps Tab** -- Table of AIPerfSweeps with per-sweep drill-down (variation curves, Pareto, children) - **Leaderboard Tab** -- Rank benchmark runs by any metric (throughput, latency percentiles, etc.) - **Compare Tab** -- Side-by-side comparison of multiple jobs to identify performance differences - **History Tab** -- Time-series charts showing how metrics evolve across all your benchmark runs See [Web Dashboard](/aiperf/dev/kubernetes/web-dashboard) for the full page-by-page reference. ### Quick Navigation Use **Ctrl+K** to open the command palette and quickly jump to any job or page. Search by job name or view recent benchmarks. --- ## Operator Prometheus Metrics The kopf operator container exposes a Prometheus `/metrics` endpoint from an in-process daemon thread (`src/aiperf/operator/metrics.py`). The port is `AIPERF_METRICS_PORT` (`OperatorEnvironment.METRICS_PORT`, default **9090**; set to `0` to disable). This is separate from the results-server on 8081. ```bash kubectl port-forward -n aiperf-system deploy/aiperf-operator 9090:9090 curl http://localhost:9090/metrics ``` Exposed series: | Metric | Type | Labels | Meaning | |---|---|---|---| | `aiperf_operator_handler_duration_seconds` | Histogram | `handler` | Wall-clock duration of each instrumented kopf reconcile handler. | | `aiperf_operator_handler_total` | Counter | `handler`, `outcome` | Reconcile-handler invocations by outcome. | | `aiperf_operator_completion_claim_races_total` | Counter | — | Lost `try_claim_completion` races (concurrent ticks contending for the completion claim). | `outcome` is one of four values: `success` (returned normally), `retry` (raised `kopf.TemporaryError`, so kopf will re-dispatch), `fatal` (raised `kopf.PermanentError`, so kopf stops retrying and the CR is stuck), and `error` (anything else, including `CancelledError`, `KeyboardInterrupt`, and `SystemExit`). `retry` and `fatal` are separated so you can alert on stuck CRs without false positives from transient apiserver hiccups. Only kopf reconcile handlers are instrumented (via the `@track_handler("name")` decorator); helper functions are not. ### Benchmark Metrics from the Controller Pod Separately from the operator's own reconcile metrics, each benchmark's controller pod serves its live benchmark metrics in Prometheus exposition format from the `api` container at `/metrics` on the API port (default 9090). The controller pod is annotated with `prometheus.io/scrape: "true"`, `prometheus.io/port`, and `prometheus.io/path: /metrics`, so an annotation-based Prometheus scrape config picks it up without extra configuration. Set `serviceMonitor.enabled=true` in the chart to have a Prometheus Operator `ServiceMonitor` scrape the Service's `metrics` port at `/metrics`. It is off by default, and is skipped when `operator.metrics.port` is `0` or when the `monitoring.coreos.com/v1` CRDs are absent. The repository ships no Grafana dashboards or `PrometheusRule` alerts. --- ## Related Documentation - [Getting Started](/aiperf/dev/kubernetes/getting-started-on-kubernetes) -- First benchmark walkthrough - [Kubernetes Configuration](/aiperf/dev/kubernetes/kubernetes-configuration-reference) -- All CRD fields and deployment options - [Production Deployments](/aiperf/dev/kubernetes/production-deployment) -- CI/CD, Kueue, and GitOps workflows