Logs

aiperf kube logs fetches or streams container logs from a specific AIPerfJob. It targets the controller pod’s service containers, the per-pod worker / record-processor containers, and the sidecars that surround them. Use it for post-mortem triage of a finished run, for live streaming during an active benchmark, or for bulk-dumping every pod’s log to disk.

Unlike aiperf kube debug, which is a one-shot diagnostic snapshot across an entire namespace, logs targets a single AIPerfJob and operates at container granularity.

CLI reference

$
aiperf kube logs [JOB_ID] [OPTIONS]

Job resolution works exactly like every other aiperf kube subcommand: if you omit JOB_ID, the CLI reads the last-benchmark cache (~/.aiperf/last_kube_benchmark.json) written by aiperf kube profile / aiperf kube sweep and reuses that ID and namespace. aiperf kube generate does not write this cache — it never contacts the cluster. Supplying --namespace on the command line always overrides the cached namespace.

Container catalog

AIPerf pods run one process per container. Most names below are among the 11 canonical container-name constants declared on aiperf.kubernetes.constants.Containers (see src/aiperf/kubernetes/constants.py); the worker-<N> and record-processor-<N> names are formed at runtime by indexing and are not declared constants. Feed any of them to --container verbatim.

Controller pod

The controller pod runs between six and nine containers — seven with stock settings. create_controller_containers always emits the five control-plane services plus results-sidecar, prepends event-bus-proxy when AIPERF_K8S_EVENT_BUS_SIDECAR_ENABLED is true (the default), and appends the two optional managers only when their features are enabled:

Worker pod

Each worker pod runs three kinds of containers. The <N> indices are pod-local and start at 0:

Deprecated name

worker-manager is listed on Containers for backwards compatibility with older manifests; it is not placed on any pod rendered by the current _JobSetManifestBuilder. Passing --container worker-manager matches nothing and emits No matching containers found.

Default behaviour (no --container)

When --container is omitted, aiperf kube logs enumerates every pod matching app=aiperf,aiperf.nvidia.com/job-id=<JOB_ID> and prints the logs of every container on every pod, in the order the API returns them. For a default JobSet with worker_replicas=N and workers_per_pod=W, this fans out to:

• the 6–9 controller-pod containers (depending on which optional managers are enabled and whether the event-bus proxy sidecar is configured), plus
• N * (1 + W + record_processors_per_pod) worker-pod containers.

For large benchmarks that total quickly runs to hundreds of containers, so you almost always want --container for interactive triage and -o <dir> for a full capture.

Follow semantics

$
aiperf kube logs -f --container control-plane

The -f/--follow flag streams logs over a long-lived read_namespaced_pod_log(follow=True) connection and prints lines to stdout as they arrive. Key properties:

• Single-target streaming. The implementation breaks out of the target loop after the first streamed container (if follow: break). If you pass -f and the resolved target list has more than one entry, the CLI prints a single-line warning of the form:

  Follow mode streams one container at a time. Showing controller-0/control-plane (9 targets total). Use --container to select a specific container.

  and only the first target is followed.

• No auto-reattach. When the underlying pod is deleted, restarted, or evicted, the stream ends and the command returns. AIPerf does not transparently reconnect; re-run the command.

• Lines are soft-wrapped, never broken mid-token. A log line wider than the terminal is folded by the terminal itself, so a long URL or stack frame stays copy-pasteable. Piping the output preserves one line per log line.

• Tail-at-start is supported. -f --tail 200 begins the stream at the last 200 buffered lines, then follows live.

• Ctrl+C terminates the stream cleanly; the finally block releases the underlying HTTP response so the connection does not leak.

Output format

Output is plain UTF-8 text, one line per log record. Each target is introduced by print_header("<pod>/<container>"), which emits a blank line, the <pod>/<container> title, and a box-drawing rule as wide as the title. Because the header goes through the aiperf.kube Rich logger, those three lines also carry the logger’s timestamp and level columns; the log lines themselves are written straight to the Rich console and carry no prefix:

20:14:07 INFO
20:14:07 INFO     controller-0/control-plane
20:14:07 INFO     ──────────────────────────
<log line 1>
<log line 2>
...

Log lines are printed with highlighting and markup disabled, so nothing in a log line is reinterpreted as Rich markup. Bytes that cannot be decoded as UTF-8 are replaced (errors="replace") so a single corrupt line never aborts the stream.

Bulk dump to disk

$
aiperf kube logs <JOB_ID> -o ./triage-2026-04-22

With -o <DIR>:

• The CLI creates <DIR>/logs/ only once the label selector has matched at least one pod. Nothing is created when it matches none.

• For every pod matching the job label, it shells out to kubectl logs -n <NAMESPACE> <POD_NAME> --all-containers=true --prefix (forwarding --kubeconfig / --context when set) and writes the stdout to <DIR>/logs/<POD_NAME>.log.

• The closing line reports what actually reached disk:

  Saved logs for 7 of 8 pod(s) to ./triage-2026-04-22/logs/

  A pod that produced no file emits its own warning first, carrying the kubectl logs exit code and stderr — for example Could not save logs -- worker-0-0: kubectl logs exited 1: container has not started. When no pod produced a file at all, the closing line is a warning (No logs written to ...), never a success.

Notes on the bulk dump path specifically:

• It writes one file per pod, not per container. Because --all-containers --prefix is always passed, every container on the pod is captured into that single file, with each line prefixed [pod/<pod>/<container>] so you can split them apart afterwards.
• It calls the local kubectl binary via run_command, so kubectl must be on PATH and able to reach the cluster with the same credentials the rest of the aiperf kube CLI uses.
• --follow, --container, and --tail are ignored in --output mode; the dump always captures the full buffered log of each pod.

Exit codes

logs follows the target-addressing exit-code convention: it narrates what it found rather than gating on it, with one exception — a target that cannot be addressed at all is an error.

This makes aiperf kube logs <ID> >/dev/null usable as a CI existence check, while a completed job whose pods have aged out still succeeds.

Troubleshooting

No pods found for <JOB_ID> in namespace <NS>

The job label selector built by job_selector in src/aiperf/kubernetes/client_selectors.py (app=aiperf,aiperf.nvidia.com/job-id=<JOB_ID>) matched nothing, but the benchmark itself still exists, so the command exits 0. Almost always the JobSet has been garbage-collected by TTL: once the pods are gone, their logs are gone from the API server too. Check the CR phase with aiperf kube list --watch, and pull the run’s captured output with aiperf kube results instead.

No AIPerf job found with ID: <JOB_ID>

Nothing named <JOB_ID> exists in the resolved namespace — no AIPerfJob, no AIPerfSweep, and no bare JobSet — so the command exits 1. Usually one of:

• The job ID is wrong (typo, stale cache). Run aiperf kube list to see live jobs.
• The CR was deleted (aiperf kube delete, or namespace teardown).
• --namespace points at the wrong namespace. Without it, the namespace comes from the cached last benchmark or your kubeconfig context.

No matching containers found

Pods exist for the job but none of them have a container named --container <NAME>. Re-check the spelling against the container catalog. Common mistakes:

• worker (not a real container name — workers are indexed: worker-0, worker-1, …).
• worker-manager (deprecated compat name, not present on any pod).
• record-processor without an index.

Error getting logs: (400) or (404)

Raised by kubernetes_asyncio when the API server rejects the pod-log request. Typical causes:

• The container has not started yet (status ContainerCreating, PodInitializing, or stuck on image pull). Use aiperf kube debug <JOB_ID> to see pod phases and events.
• The pod was deleted between the list and the read call (race between follow reconnects and TTL cleanup).
• The container terminated and its log buffer was rotated out of the kubelet cache — try --previous-style recovery via kubectl logs --previous against the pod name directly.

RBAC denials

Log reads require pods/log access in the target namespace. If the CLI prints Error getting logs: (403), confirm your kubeconfig user or ServiceAccount has the verb; see rbac-security.md for the roles the operator itself uses and a minimal user-side role for log access.

Empty output

A container can legitimately produce zero log lines — for example, a short-lived worker that failed its startup probe before any handler ran. In follow mode the stream simply stays silent until the container writes or exits; in buffered mode the CLI prints a single empty line. Cross-check with aiperf kube debug and with container status (Waiting reason / exit code) to rule out a crashed pod versus a quiet one.

Related commands

• aiperf kube debug — one-shot namespace-wide snapshot including pod phases, events, and recent log tails.
• aiperf kube attach — stream live benchmark progress over the controller’s WebSocket instead of container logs.
• aiperf kube list — list live AIPerfJobs to discover the ID you want to pass here.