> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/aiperf/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/aiperf/_mcp/server. # Logs `aiperf kube logs` fetches or streams container logs from a specific AIPerfJob. It targets the controller pod's service containers, the per-pod worker / record-processor containers, and the sidecars that surround them. Use it for post-mortem triage of a finished run, for live streaming during an active benchmark, or for bulk-dumping every pod's log to disk. Unlike `aiperf kube debug`, which is a one-shot diagnostic snapshot across an entire namespace, `logs` targets a single AIPerfJob and operates at container granularity. --- ## CLI reference ```bash aiperf kube logs [JOB_ID] [OPTIONS] ``` | Flag / argument | Type | Default | Description | |---|---|---|---| | `JOB_ID` (positional) | `str` | last-deployed job | AIPerfJob ID — or an `AIPerfSweep` name, when combined with `-v` — to pull logs from. If omitted, falls back to the job ID stored on disk by the most recent `aiperf kube profile` / `aiperf kube sweep` invocation. | | `--container` | `str` | *(all containers on every pod)* | Restrict to a single container name. See the [container catalog](#container-catalog) below. | | `-f`, `--follow` | flag | `false` | Stream logs in real time. Only one target can be followed at a time; see [follow semantics](#follow-semantics). | | `--tail` | `int` | *(full log)* | Return only the last N lines. Combine with `-f` to start a stream at the tail. | | `-o`, `--output` | `Path` | *(stdout)* | Write logs to a directory instead of printing. See [bulk dump](#bulk-dump-to-disk). | | `-v`, `--variation` | `int` | unset | When `JOB_ID` names an `AIPerfSweep`, the child variation index (`0`..`199`). Resolves to `-v`. | | `-t`, `--trial` | `int` | unset | Trial index (`0`..`9`) within a sweep variation, resolving to `-v-t`. Requires `-v`. | | `--ignore-not-found` | flag | `false` | Exit `0` instead of `1` when the target benchmark does not exist. Mirrors `kubectl`'s flag of the same name. See [exit codes](#exit-codes). | | `-n`, `--namespace` | `str` | cached last-benchmark namespace, else the kubeconfig context namespace | Namespace that holds the AIPerfJob. Composite flag from `KubeManageOptions`. | | `--kubeconfig` | `str` | `$KUBECONFIG` / `~/.kube/config` | kubeconfig file to use. Composite flag from `KubeManageOptions`. | | `--kube-context` | `str` | current context | kubeconfig context to use. Composite flag from `KubeManageOptions`. | Job resolution works exactly like every other `aiperf kube` subcommand: if you omit `JOB_ID`, the CLI reads the last-benchmark cache (`~/.aiperf/last_kube_benchmark.json`) written by `aiperf kube profile` / `aiperf kube sweep` and reuses that ID and namespace. `aiperf kube generate` does **not** write this cache — it never contacts the cluster. Supplying `--namespace` on the command line always overrides the cached namespace. --- ## Container catalog AIPerf pods run one process per container. Most names below are among the 11 canonical container-name constants declared on `aiperf.kubernetes.constants.Containers` (see `src/aiperf/kubernetes/constants.py`); the `worker-` and `record-processor-` names are formed at runtime by indexing and are **not** declared constants. Feed any of them to `--container` verbatim. ### Controller pod The controller pod runs between six and nine containers — seven with stock settings. `create_controller_containers` always emits the five control-plane services plus `results-sidecar`, prepends `event-bus-proxy` when `AIPERF_K8S_EVENT_BUS_SIDECAR_ENABLED` is true (the default), and appends the two optional managers only when their features are enabled: | Container | Purpose | |---|---| | `control-plane` | SystemController and orchestration logic. | | `event-bus-proxy` | XPUB/XSUB event-bus proxy sidecar. Isolates pub/sub forwarding from the SystemController event loop so large fan-ins of workers and record processors at startup do not starve the control plane. | | `dataset-manager` | Dataset generation and memory-map serving. | | `timing-manager` | Request scheduling and credit timing. | | `records-manager` | Metric record aggregation, export, and storage. | | `api` | HTTP + WebSocket API surface for monitoring, progress, and result artifacts. | | `results-sidecar` | Lightweight sidecar serving exported artifacts from `/results`; outlives the main controller on export so the operator can still download results. | | `gpu-telemetry-manager` | DCGM GPU metrics collection. Present only when GPU telemetry is enabled. | | `server-metrics-manager` | Prometheus server-side metrics scraping. Present only when server metrics are enabled. | ### Worker pod Each worker pod runs three kinds of containers. The `` indices are pod-local and start at `0`: | Container | Purpose | |---|---| | `worker-group-manager` | Per-pod worker-group lifecycle, dataset download, local inference proxy, raw-record upload coordination. One per pod. | | `worker-` | One request-issuing worker. Count controlled by `workers_per_pod`. | | `record-processor-` | One record processor. Count defaults to `workers_per_pod / RECORD_PROCESSOR_SCALE_FACTOR`. | ### Deprecated name `worker-manager` is listed on `Containers` for backwards compatibility with older manifests; it is **not** placed on any pod rendered by the current `_JobSetManifestBuilder`. Passing `--container worker-manager` matches nothing and emits `No matching containers found`. --- ## Default behaviour (no `--container`) When `--container` is omitted, `aiperf kube logs` enumerates every pod matching `app=aiperf,aiperf.nvidia.com/job-id=` and prints the logs of **every container on every pod**, in the order the API returns them. For a default JobSet with `worker_replicas=N` and `workers_per_pod=W`, this fans out to: - the 6–9 controller-pod containers (depending on which optional managers are enabled and whether the event-bus proxy sidecar is configured), plus - `N * (1 + W + record_processors_per_pod)` worker-pod containers. For large benchmarks that total quickly runs to hundreds of containers, so you almost always want `--container` for interactive triage and `-o ` for a full capture. --- ## Follow semantics ```bash aiperf kube logs -f --container control-plane ``` The `-f/--follow` flag streams logs over a long-lived `read_namespaced_pod_log(follow=True)` connection and prints lines to stdout as they arrive. Key properties: - **Single-target streaming.** The implementation breaks out of the target loop after the first streamed container (`if follow: break`). If you pass `-f` and the resolved target list has more than one entry, the CLI prints a single-line warning of the form: ```text Follow mode streams one container at a time. Showing controller-0/control-plane (9 targets total). Use --container to select a specific container. ``` and only the first target is followed. - **No auto-reattach.** When the underlying pod is deleted, restarted, or evicted, the stream ends and the command returns. AIPerf does not transparently reconnect; re-run the command. - **Lines are soft-wrapped, never broken mid-token.** A log line wider than the terminal is folded by the terminal itself, so a long URL or stack frame stays copy-pasteable. Piping the output preserves one line per log line. - **Tail-at-start is supported.** `-f --tail 200` begins the stream at the last 200 buffered lines, then follows live. - **Ctrl+C** terminates the stream cleanly; the `finally` block releases the underlying HTTP response so the connection does not leak. --- ## Output format Output is plain UTF-8 text, one line per log record. Each target is introduced by `print_header("/")`, which emits a blank line, the `/` title, and a box-drawing rule as wide as the title. Because the header goes through the `aiperf.kube` Rich logger, those three lines also carry the logger's timestamp and level columns; the log lines themselves are written straight to the Rich console and carry no prefix: ```text 20:14:07 INFO 20:14:07 INFO controller-0/control-plane 20:14:07 INFO ────────────────────────── ... ``` Log lines are printed with highlighting and markup disabled, so nothing in a log line is reinterpreted as Rich markup. Bytes that cannot be decoded as UTF-8 are replaced (`errors="replace"`) so a single corrupt line never aborts the stream. --- ## Bulk dump to disk ```bash aiperf kube logs -o ./triage-2026-04-22 ``` With `-o `: - The CLI creates `/logs/` only once the label selector has matched at least one pod. Nothing is created when it matches none. - For every pod matching the job label, it shells out to `kubectl logs -n --all-containers=true --prefix` (forwarding `--kubeconfig` / `--context` when set) and writes the stdout to `/logs/.log`. - The closing line reports what actually reached disk: ```text Saved logs for 7 of 8 pod(s) to ./triage-2026-04-22/logs/ ``` A pod that produced no file emits its own warning first, carrying the `kubectl logs` exit code and stderr — for example `Could not save logs -- worker-0-0: kubectl logs exited 1: container has not started`. When no pod produced a file at all, the closing line is a warning (`No logs written to ...`), never a success. Notes on the bulk dump path specifically: - It writes **one file per pod**, not per container. Because `--all-containers --prefix` is always passed, every container on the pod is captured into that single file, with each line prefixed `[pod//]` so you can split them apart afterwards. - It calls the local `kubectl` binary via `run_command`, so `kubectl` must be on `PATH` and able to reach the cluster with the same credentials the rest of the `aiperf kube` CLI uses. - `--follow`, `--container`, and `--tail` are **ignored** in `--output` mode; the dump always captures the full buffered log of each pod. --- ## Exit codes `logs` follows the [target-addressing exit-code convention](/aiperf/dev/kubernetes/end-to-end-workflow#exit-code-convention-for-target-addressing-commands): it narrates what it found rather than gating on it, with one exception — a target that cannot be addressed at all is an error. | Scenario | Exit code | |---|---| | Logs printed or written | `0` | | Pods matched but a container produced no output | `0` | | Job exists, but its pods have been garbage-collected | `0` | | `--container` matched no container on any pod | `0` | | No AIPerfJob, AIPerfSweep or JobSet with that name exists | `1` | | No `JOB_ID` given and no last-benchmark record on disk | `1` | | Any of the above two, with `--ignore-not-found` | `0` | This makes `aiperf kube logs >/dev/null` usable as a CI existence check, while a completed job whose pods have aged out still succeeds. --- ## Troubleshooting ### `No pods found for in namespace ` The job label selector built by `job_selector` in `src/aiperf/kubernetes/client_selectors.py` (`app=aiperf,aiperf.nvidia.com/job-id=`) matched nothing, but the benchmark itself still exists, so the command exits `0`. Almost always the JobSet has been garbage-collected by TTL: once the pods are gone, their logs are gone from the API server too. Check the CR phase with `aiperf kube list --watch`, and pull the run's captured output with `aiperf kube results` instead. ### `No AIPerf job found with ID: ` Nothing named `` exists in the resolved namespace — no AIPerfJob, no AIPerfSweep, and no bare JobSet — so the command exits `1`. Usually one of: - The job ID is wrong (typo, stale cache). Run `aiperf kube list` to see live jobs. - The CR was deleted (`aiperf kube delete`, or namespace teardown). - `--namespace` points at the wrong namespace. Without it, the namespace comes from the cached last benchmark or your kubeconfig context. ### `No matching containers found` Pods exist for the job but none of them have a container named `--container `. Re-check the spelling against the [container catalog](#container-catalog). Common mistakes: - `worker` (not a real container name — workers are indexed: `worker-0`, `worker-1`, ...). - `worker-manager` (deprecated compat name, not present on any pod). - `record-processor` without an index. ### `Error getting logs: (400)` or `(404)` Raised by `kubernetes_asyncio` when the API server rejects the pod-log request. Typical causes: - The container has not started yet (status `ContainerCreating`, `PodInitializing`, or stuck on image pull). Use `aiperf kube debug ` to see pod phases and events. - The pod was deleted between the list and the read call (race between `follow` reconnects and TTL cleanup). - The container terminated and its log buffer was rotated out of the kubelet cache — try `--previous`-style recovery via `kubectl logs --previous` against the pod name directly. ### RBAC denials Log reads require `pods/log` access in the target namespace. If the CLI prints `Error getting logs: (403)`, confirm your kubeconfig user or ServiceAccount has the verb; see [rbac-security.md](/aiperf/dev/kubernetes/rbac-and-security) for the roles the operator itself uses and a minimal user-side role for log access. ### Empty output A container can legitimately produce zero log lines — for example, a short-lived worker that failed its startup probe before any handler ran. In follow mode the stream simply stays silent until the container writes or exits; in buffered mode the CLI prints a single empty line. Cross-check with `aiperf kube debug` and with container status (`Waiting` reason / exit code) to rule out a crashed pod versus a quiet one. --- ## Related commands - [`aiperf kube debug`](/aiperf/dev/kubernetes/debug-command) — one-shot namespace-wide snapshot including pod phases, events, and recent log tails. - [`aiperf kube attach`](/aiperf/dev/kubernetes/attach-to-a-running-benchmark) — stream live benchmark progress over the controller's WebSocket instead of container logs. - [`aiperf kube list`](/aiperf/dev/kubernetes/end-to-end-workflow) — list live AIPerfJobs to discover the ID you want to pass here.