Results Server API

The AIPerf operator ships a standalone HTTP server (the results server) as a sidecar inside the operator pod. It exposes a catalog of /api/v1/... endpoints for listing jobs, downloading raw result files, querying the SQLite-backed runs index for analytics, and introspecting live AIPerfJob state.

This reference documents the primary endpoints served by that process; it is not exhaustive. The results server also mounts a sweeps router (/api/v1/sweeps/... — list, create, detail, cells, children, epochs, config, events, logs, artifacts), a config router (/api/v1/config/retention, /api/v1/config/features), a validate router (POST /api/v1/validate), additional per-job routes (/api/v1/jobs/{ns}/{name}/epochs|events|logs, zip and per-epoch result downloads), and an admin /admin/index/run/{ns}/{job_id} index-row route. The source of truth is src/aiperf/operator/results_server.py and the routers under src/aiperf/operator/routers/.

How to reach the API

The results server listens on resultsServer.port (default 8081) inside the operator pod. Helm injects this as AIPERF_RESULTS_SERVER_PORT, which the server validates as an integer from 1 through 65535. It is fronted by the operator’s Service, not exposed externally by default. Two common access patterns:

Port-forward via the CLI

$
aiperf kube dashboard

This opens the browser UI and port-forwards the results server to an ephemeral local port. Pass --no-browser to just print the URL, or --port 8081 to pin a specific local port. The port-forward stays open until Ctrl+C.

Direct kubectl port-forward

$
kubectl -n aiperf-system port-forward svc/aiperf-operator 8081:8081

The base URL is then http://localhost:8081, and all endpoints below are reachable at http://localhost:8081/api/v1/....

Request topology

The sidecar reads result files from the shared PVC and queries the Kubernetes API (via in-cluster RBAC) for live job/cluster state. Read-only routes have no per-request authentication layer; mutating routes are disabled by default and require explicit bearer-token configuration when enabled. See Auth / security.

Endpoint reference

Final artifacts are run-scoped, not job-scoped: the three job-scoped /api/v1/results/{namespace}/{job_id}[.zip|/{filename}] routes exist only to reject with 409 and point the caller at /runs/<epoch>/.... This keeps a caller from pairing a live job status with whichever run latest.txt happened to point at mid-request.

The per-run file-list response includes per_record_filename and server_metrics_filename when those artifacts are present. Clients should use these fields rather than assuming the default filenames: artifacts.prefix changes both names for that particular run.

GET /api/v1/jobs and GET /api/v1/jobs/{namespace}/{name} are served through an ETag helper: they set ETag plus Cache-Control: no-cache and answer a matching If-None-Match with 304 Not Modified, so a polling client pays only the hash on an unchanged job list.

Meta

GET /healthz

Liveness probe. Always returns 200 OK.

$
curl http://localhost:8081/healthz

1
{"status": "ok"}

POST /api/v1/validate

Dry-run validation of an AIPerfJob or AIPerfSweep manifest. Nothing is created and no cluster access is required, so this route is not behind the mutating-route token.

Request body

Unknown top-level body fields are rejected (extra="forbid" → 422).

$
curl -X POST http://localhost:8081/api/v1/validate \
>
  -H 'Content-Type: application/json' \
>
  -d '{"manifest": {"apiVersion": "aiperf.nvidia.com/v1alpha1", "kind": "AIPerfJob", "spec": {}}}'

1
{
2
  "passed": false,
3
  "errors": ["spec.benchmark: field required"],
4
  "warnings": []
5
}

Because the route is unauthenticated and exists purely to convert a bad manifest into a structured verdict, a well-formed request body always answers 200 OK with this shape — never 500. A validator that raises unexpectedly is caught, logged server-side, and reported as passed: false with a single Validation aborted: <ExceptionType>: <message> entry in errors. Only a malformed request body (bad JSON, missing manifest, unknown top-level field) produces a non-200, as 422.

Jobs

Live state read directly from the Kubernetes API. Every endpoint in this section returns 503 Service Unavailable if the results server could not initialize its Kubernetes client at startup (e.g. no kubeconfig and not running in-cluster).

The one exception is GET /api/v1/jobs/{namespace}/{name}/epochs, which takes the API client as optional: without a cluster connection it still serves the PVC-derived epoch list, just with status left at "unknown" for epochs the runs index has not ingested.

GET /api/v1/jobs

List every known AIPerfJob across all namespaces.

The response is a union, not just live CRs: job_union.list_all_jobs merges live AIPerfJob CRs (source: "live"), PVC-only historical runs whose CRs were garbage-collected (source: "archived"), and runs present in both (source: "both"). Entries are keyed by (namespace, name); overlaps prefer CR values on live fields and backfill historical fields from the PVC.

$
curl http://localhost:8081/api/v1/jobs

1
{
2
  "jobs": [
3
    {
4
      "name": "aiperf-bench-7f2a",
5
      "namespace": "my-benchmarks",
6
      "phase": "Running",
7
      "jobId": "aiperf-bench-7f2a",
8
      "created": "2026-04-22T14:08:11Z",
9
      "source": "live"
10
    }
11
  ]
12
}

AIPerfJobInfo (kubernetes/models.py) is a camelCase-aliased model, so live metric fields serialize as throughputRps, latencyP99Ms, ttftMs, outputTokenThroughputTps, interTokenLatencyMs, progressPercent, totalRequests, errorRate, and the sweep-child fields as sweepName, variationIndex, variationLabel, variationValues, trialIndex. All are optional and serialize as null when the CR status has not reported them.

Status codes

• 200 — success (possibly empty list)
• 304 — If-None-Match matched the current ETag
• 401 / 403 — surfaced verbatim if the sidecar’s ServiceAccount lacks RBAC to list aiperfjobs.aiperf.nvidia.com
• 503 — Kubernetes client unavailable

GET /api/v1/jobs/{namespace}/{name}

Fetch a single AIPerfJob CR plus its pod roster. Accepts an optional ?epoch= query parameter to pin the archived half to a specific historical run directory instead of following latest.txt.

Archived (PVC-only) jobs have no cluster CR, so the response carries the archived summary with an empty status object and an empty pods list.

Path parameters

$
curl http://localhost:8081/api/v1/jobs/my-benchmarks/aiperf-bench-7f2a

1
{
2
  "job": {
3
    "name": "aiperf-bench-7f2a",
4
    "namespace": "my-benchmarks",
5
    "phase": "Running"
6
  },
7
  "status": {
8
    "phase": "Running",
9
    "conditions": [],
10
    "liveMetrics": {"requestThroughput": 1842.3}
11
  },
12
  "pods": [
13
    {"name": "aiperf-bench-7f2a-controller-0", "phase": "Running", "ready": true, "restarts": 0, "containers": ["controller"]},
14
    {"name": "aiperf-bench-7f2a-worker-0",     "phase": "Running", "ready": true, "restarts": 0, "containers": ["worker"]}
15
  ]
16
}

Pods are filtered by the label selector aiperf.nvidia.com/job-id=<name>. Each entry’s containers list is the pod spec’s container names in spec order — those are exactly the accepted values for ?container= on the logs endpoint.

Status codes

• 200 — success
• 304 — If-None-Match matched the current ETag
• 400 — namespace or name is not a valid RFC 1123 name (rejected before any path join), or ?epoch= does not match the epoch allowlist
• 404 — no AIPerfJob with that name in that namespace and no archived run on the PVC
• 401 / 403 — RBAC denial
• 503 — Kubernetes client unavailable

POST /api/v1/jobs

Create an AIPerfJob CR from a manifest dict shaped exactly like kubectl apply -f input. Mutating route — see Auth / security.

Request body

$
curl -X POST \
>
  -H "Authorization: Bearer ${AIPERF_OPERATOR_MUTATING_ROUTES_TOKEN}" \
>
  -H 'Content-Type: application/json' \
>
  -d @- http://localhost:8081/api/v1/jobs <<'EOF'
$
{"manifest": {"metadata": {"name": "aiperf-bench-7f2a", "namespace": "my-benchmarks"}, "spec": {}}}
$
EOF

1
{
2
  "namespace": "my-benchmarks",
3
  "name": "aiperf-bench-7f2a",
4
  "uid": "9c1e6f3a-1c2b-4f4e-9a0f-7bb4a3d21b55"
5
}

Status codes

• 201 — CR created
• 400 — manifest is not an object, metadata is not an object, or metadata.name is missing
• 401 / 403 — mutating-route gate or RBAC denial
• 409 — a CR with that name already exists in that namespace
• 422 — apiserver rejected the manifest against the CRD schema
• 503 — Kubernetes client unavailable

POST /api/v1/jobs/{namespace}/{name}/cancel

Request cancellation of a running benchmark by patching the CR’s spec.cancel to true.

This endpoint is asynchronous. It returns immediately after the patch; the kopf operator observes the change and drives workers to a stopped state over the next several seconds. Poll GET /api/v1/jobs/{namespace}/{name} and wait for status.phase to become Cancelled, Failed, or Succeeded if you need to confirm termination.

$
curl -X POST \
>
  -H "Authorization: Bearer ${AIPERF_OPERATOR_MUTATING_ROUTES_TOKEN}" \
>
  http://localhost:8081/api/v1/jobs/my-benchmarks/aiperf-bench-7f2a/cancel

1
{"cancelled": true}

Status codes

• 200 — patch submitted
• 400 — the job is archived-only (PVC results exist but the CR is gone, so there is nothing to patch)
• 404 — neither a live CR nor a PVC directory exists for that namespace/name
• 401 / 403 — RBAC denial
• 409 — concurrent-modification conflict (retry)
• 503 — Kubernetes client unavailable

GET /api/v1/cluster

Best-effort cluster-wide totals for the dashboard header.

$
curl http://localhost:8081/api/v1/cluster

1
{
2
  "nodes": 12,
3
  "gpus": 96,
4
  "gpus_used": 40,
5
  "gpus_free": 56,
6
  "utilization_percent": 41.7,
7
  "gpu_nodes": 8,
8
  "nodes_free": 4,
9
  "nodes_partial": 3,
10
  "nodes_full": 1,
11
  "kubernetes_version": "v1.29.4",
12
  "cluster_name": "dgx-prod"
13
}

ClusterResponse (operator/routers/jobs_models.py) carries the full GPU accounting (gpus_used/gpus_free/utilization_percent), GPU-node breakdown (gpu_nodes, nodes_free/nodes_partial/nodes_full), and an optional cluster_name in addition to the basic nodes/gpus/kubernetes_version. Both the node list and version query are best-effort: if RBAC is insufficient or the call fails, kubernetes_version is reported as "unknown" and counts fall back to 0. The endpoint does not surface errors for these sub-queries.

WS /api/v1/jobs/{namespace}/{name}/ws

Per-job WebSocket proxy. The browser dashboard’s per-job detail page uses this to subscribe to the same realtime message stream the controller pod publishes (e.g. realtime_metrics, credit_phase_progress, worker_group_stats), so KPI tiles update at the controller’s emit cadence (~1Hz) instead of the page’s REST poll interval.

The proxy is transparent: it does not subscribe on the client’s behalf. After the WS opens, the browser sends the controller’s standard subscribe frame:

1
{"type": "subscribe", "message_types": ["realtime_metrics"]}

…and from then on receives upstream frames verbatim. Implementation: src/aiperf/operator/routers/jobs_ws.py.

Topology

The operator looks up the AIPerfJob CR’s status.jobSetName, derives the controller pod’s headless-service DNS via controller_dns_name(jobset_name, namespace), and opens an aiohttp WebSocket to ws://<controller-dns>:<API_SERVICE>/ws. Two asyncio pumps then bridge frames in both directions until either side closes.

Refusal close codes

The proxy refuses connections with private-use (4xxx) WebSocket close codes so the browser can distinguish causes:

Results (file serving)

All file-serving endpoints read from the shared results PVC mounted at AIPERF_RESULTS_DIR (default /data). Files are laid out as <namespace>/<job_id>/<epoch>/<filename>, where <epoch> is a decimal-seconds run directory and a sibling latest.txt names the newest one.

GET /api/v1/results

List every namespace/job directory with at least one stored file.

$
curl http://localhost:8081/api/v1/results

1
{
2
  "jobs": [
3
    {
4
      "namespace": "my-benchmarks",
5
      "job_id": "aiperf-bench-7f2a",
6
      "file_count": 8,
7
      "total_size_bytes": 24837211,
8
      "model": "meta-llama/Llama-3.1-70B",
9
      "endpoint": "http://llama:8000/v1/chat/completions"
10
    }
11
  ]
12
}

model and endpoint are lifted from the run’s persisted job_spec.json and are null when that file is missing or unreadable. Returns an empty jobs list (not a 404) if the PVC base directory doesn’t exist yet.

GET /api/v1/results/{namespace}/{job_id}/runs

List every stored run epoch for one job, newest first, with latest_epoch naming the current latest.txt target (null when the job has no runs yet).

$
curl http://localhost:8081/api/v1/results/my-benchmarks/aiperf-bench-7f2a/runs

GET /api/v1/results/{namespace}/{job_id}/runs/{epoch}

List all result files for one run epoch.

$
curl http://localhost:8081/api/v1/results/my-benchmarks/aiperf-bench-7f2a/runs/1777472025

1
{
2
  "namespace": "my-benchmarks",
3
  "job_id": "aiperf-bench-7f2a",
4
  "ready": true,
5
  "summary_available": true,
6
  "per_record_filename": "profile_export.jsonl",
7
  "server_metrics_filename": "server_metrics.json",
8
  "files": [
9
    {
10
      "name": "profile_export_aiperf.csv",
11
      "stored_name": "profile_export_aiperf.csv.zst",
12
      "size_bytes": 381220,
13
      "compressed": true,
14
      "mtime_epoch": 1777472025
15
    },
16
    {
17
      "name": "inputs.json",
18
      "stored_name": "inputs.json",
19
      "size_bytes": 4812,
20
      "compressed": false,
21
      "mtime_epoch": 1777472025
22
    }
23
  ]
24
}

The name field is the display name (zstd suffix stripped); use it as the {filename} path parameter on the download endpoint. stored_name is the actual file on disk. ready reflects the sidecar’s readiness marker; summary_available says whether the profile_export quick-export alias will resolve for this run.

Status codes

• 200 — success
• 400 — namespace / job_id failed RFC 1123 path-parameter validation (this is how encoded traversal attempts are rejected)
• 404 — no run directory <namespace>/<job_id>/<epoch>/ exists
• 422 — {epoch} does not match the epoch allowlist

GET /api/v1/results/{namespace}/{job_id}/runs/{epoch}/{filename}

Download a single result file. The server handles content negotiation automatically based on Accept-Encoding.

The lookup tries <filename>.zst first, then <filename> as-is. namespace and job_id are validated as RFC 1123 names before any path join (400 on failure), and the resolved file path must stay under the run directory. Final artifacts are gated on the readiness marker: requesting one before the sidecar has written that marker returns 404 with a “Results not ready” message. Checkpoint artifacts are exempt from the gate, and the marker file itself is not downloadable.

{filename} is a path-style parameter, so files inside the run’s allowlisted subdirectories (e.g. checkpoints) are addressable with their relative path — exactly the name the listing endpoint returned.

Content negotiation for stored .zst files

Content negotiation for stored raw files

The common.compression.select_encoding helper picks the best encoding the client accepts (default IDENTITY). Content-Encoding is set only if the server is recompressing; otherwise it’s omitted.

Response headers (both paths)

• Content-Disposition: attachment; filename="<display-name>" (switched to the RFC 5987 filename*=UTF-8''… form for non-ASCII names)
• X-Filename: <display-name>
• Cache-Control: no-store and Pragma: no-cache
• Content-Type derived from the display name’s suffix: text/csv, application/json, application/x-ndjson, application/vnd.apache.parquet, image/png, else application/octet-stream

$
# Native zstd — smallest over the wire
$
curl -H "Accept-Encoding: zstd" \
>
  http://localhost:8081/api/v1/results/my-benchmarks/aiperf-bench-7f2a/runs/1777472025/profile_export_aiperf.csv \
>
  --output profile.csv.zst
$
$
# Let curl transparently decompress gzip
$
curl --compressed \
>
  http://localhost:8081/api/v1/results/my-benchmarks/aiperf-bench-7f2a/runs/1777472025/profile_export_aiperf.csv \
>
  -o profile.csv

Status codes

• 200 — stream begins (note: errors mid-stream surface as truncated bodies, not HTTP errors)
• 400 — invalid namespace / job_id path parameter
• 404 — run directory missing, results not yet marked ready, or neither <filename> nor <filename>.zst found
• 422 — {epoch} does not match the epoch allowlist

GET /api/v1/results/{namespace}/{job_id}/runs/{epoch}.zip

Stream the whole run epoch as a zip bundle, Content-Disposition-named after the run. Also gated on the readiness marker (404 before it exists).

GET /api/v1/results/{namespace}/{job_id}/runs/{epoch}/profile_export

Quick-export alias for that run’s summary JSON, so a caller does not need to know whether the artifact prefix changed the filename. Takes one query parameter, format, constrained to json (it exists so csv/parquet shortcuts can be added without a new route).

Analytics (runs-index backed)

All analytics endpoints return 503 with message "Analytics engine not initialized" if the results-server lifespan hook has not yet populated the DB handle. The exception is /analytics/scatter, which checks runs_index.is_open() itself and returns an empty entries list rather than an error, so a cold dashboard renders an empty chart instead of a failure banner.

Analytics leaderboard and history queries support the six indexed comparison metrics (request_throughput, request_latency, time_to_first_token, output_token_throughput, output_token_throughput_per_user, and inter_token_latency) and avg, p50, or p99 statistics. Unsupported metric/stat combinations return an empty response without scanning artifact summaries.

/analytics/leaderboard, /history, /compare, and /summary are backed by the runs_index SQLite store (operator/runs_index.py, exposed through the thin ResultsDB facade in operator/results_db.py) — flat-column SELECTs against indexed rows, with a zstd-compressed metrics_json blob for full-summary access. The cold-start cost (one PVC walk) moves to the operator’s startup bootstrap; subsequent supported queries use the index and retain disk fallback + lazy backfill when the index is stale.

GET /api/v1/analytics/leaderboard

Rank every run by a metric.

Query parameters

$
curl "http://localhost:8081/api/v1/analytics/leaderboard?metric=request_throughput&stat=avg&limit=5"

1
{
2
  "metric": "request_throughput",
3
  "stat": "avg",
4
  "order": "desc",
5
  "entries": [
6
    {
7
      "namespace": "my-benchmarks",
8
      "job_id": "aiperf-bench-7f2a",
9
      "epoch": "1777472025",
10
      "value": 1842.3,
11
      "unit": "requests/sec",
12
      "start_time": "2026-04-22T14:08:11Z",
13
      "end_time":   "2026-04-22T14:13:45Z",
14
      "model": "meta-llama/Llama-3.1-70B",
15
      "endpoint": "http://llama:8000/v1/chat/completions"
16
    }
17
  ]
18
}

Every entry field except namespace and job_id is nullable. The dashboard’s leaderboard page requests limit=1000 and filters client-side, so a caller that relies on the 20 default will see a shorter list than the UI does.

GET /api/v1/analytics/history

Return metric values over time, optionally filtered by model or endpoint.

Query parameters

$
curl "http://localhost:8081/api/v1/analytics/history?metric=request_latency&stat=p99&model=Llama&limit=50"

1
{
2
  "metric": "request_latency",
3
  "stat": "p99",
4
  "entries": [
5
    {
6
      "namespace": "my-benchmarks",
7
      "job_id": "aiperf-bench-7f2a",
8
      "epoch": "1777472025",
9
      "value": 412.7,
10
      "unit": "ms",
11
      "start_time": "2026-04-22T14:08:11Z",
12
      "model": "meta-llama/Llama-3.1-70B",
13
      "endpoint": "http://llama:8000/v1/chat/completions"
14
    }
15
  ]
16
}

HistoryEntry carries no end_time — that field exists only on the leaderboard rows.

GET /api/v1/analytics/scatter

Return all four dashboard scatter metrics for every indexed run in a single SQLite query, replacing the N+1 leaderboard-plus-summary fan-out the dashboard used to issue. Takes no parameters.

$
curl http://localhost:8081/api/v1/analytics/scatter

1
{
2
  "entries": [
3
    {
4
      "namespace": "my-benchmarks",
5
      "job_id": "aiperf-bench-7f2a",
6
      "epoch": "1777472025",
7
      "model": "meta-llama/Llama-3.1-70B",
8
      "request_throughput_avg": 1842.3,
9
      "request_latency_p99": 412.7,
10
      "time_to_first_token_avg": 38.1,
11
      "output_token_throughput_avg": 21504.6
12
    }
13
  ]
14
}

Entries are ordered newest epoch first. Every metric field is nullable. If the runs index is closed or the query raises, the endpoint returns {"entries": []} with 200 rather than an error.

GET /api/v1/analytics/compare

Pull a side-by-side comparison of named jobs across a set of metrics. The response pivots the runs-index rows into (metric, stat, unit, values={namespace/job_id: value}) entries for the UI.

Query parameters

$
curl "http://localhost:8081/api/v1/analytics/compare?jobs=aiperf-bench-7f2a&jobs=aiperf-bench-9d4c&metrics=request_throughput&metrics=request_latency"

1
{
2
  "job_ids": ["aiperf-bench-7f2a", "aiperf-bench-9d4c"],
3
  "metrics": ["request_throughput", "request_latency"],
4
  "entries": [
5
    {
6
      "metric": "request_throughput",
7
      "stat": "avg",
8
      "unit": "requests/sec",
9
      "values": {
10
        "my-benchmarks/aiperf-bench-7f2a": 1842.3,
11
        "my-benchmarks/aiperf-bench-9d4c": 1790.8
12
      }
13
    }
14
  ]
15
}

For each metric, the server emits one entry per stat (avg, p50, p99), plus a meta block alongside entries. Entries where no job has a value are omitted. Value keys are namespace/job_id when a namespace is known, otherwise just the job ID.

Passing a bare job name that matches runs in more than one namespace returns 409 with the ambiguous candidates listed; re-request using namespace/job syntax.

GET /api/v1/analytics/summary/{namespace}/{job_id}

Return the full aggregated summary for one job (as a raw JSON object — no Pydantic schema because the shape is driven by the metrics plugin registry). Accepts an optional ?epoch= query parameter; omitted, it follows latest.txt. Endpoint credentials in the embedded input config are redacted before the body is returned.

$
curl http://localhost:8081/api/v1/analytics/summary/my-benchmarks/aiperf-bench-7f2a

Status codes

• 200 — summary found
• 400 — namespace / job_id failed RFC 1123 path-parameter validation
• 404 — no summary data for that namespace/job_id

GET /api/v1/index

Return the full job index used for fast lookups, keyed by <namespace>/<job_id>. It is backed by the runs-index SQLite store via get_db().index_entries() (operator/results_db.py / operator/runs_index.py) — there is no aiperf.operator.job_index module. The shape is a dict of index rows consumed by the dashboard.

$
curl http://localhost:8081/api/v1/index

GET /api/v1/config/{namespace}/{job_id}

Return the original CR spec/config used to run a job. The server tries four sources in order (first hit wins) and records which one it used:

1. Index (source: "index") — fast path, served from the runs-index SQLite cache.
2. Standalone spec file (source: "file") — <base>/<namespace>/<job_id>/job_spec.json on the PVC.
3. Summary extraction (source: "summary") — pulls input_config out of the aggregated summary if the spec wasn’t persisted separately.
4. Live CR (source: "cr") — fetches spec from the apiserver, covering running jobs whose artifacts haven’t been persisted yet. Skipped entirely when ?epoch= is set, since a pinned historical epoch must not be answered with the current CR.

Specs from the first three sources are passed through redact_endpoint_spec (aiperf.common.redact) before being returned.

Accepts an optional ?epoch= query param (default: follow latest).

$
curl http://localhost:8081/api/v1/config/my-benchmarks/aiperf-bench-7f2a

1
{
2
  "source": "file",
3
  "spec": {
4
    "benchmark": {
5
      "model": ["meta-llama/Llama-3.1-70B"],
6
      "endpoint": {"url": "http://llama:8000/v1/chat/completions"}
7
    }
8
  }
9
}

Status codes

• 200 — config found via one of the four sources
• 400 — namespace / job_id failed RFC 1123 path-parameter validation
• 404 — none of the sources had data for that namespace/job_id

Auth / security

Read-only results-server routes do not authenticate individual HTTP requests. Mutating routes are disabled by default and require an explicit bearer token when enabled.

• Mutating-route gate. POST /api/v1/jobs, POST /api/v1/jobs/{namespace}/{name}/cancel, and POST /api/v1/sweeps return 403 unless AIPERF_OPERATOR_MUTATING_ROUTES_ENABLED=true is set on the results-server. When enabled, AIPERF_OPERATOR_MUTATING_ROUTES_TOKEN must also be set — an enabled-but-tokenless server also returns 403 — and callers must send Authorization: Bearer <token>; a missing, non-bearer, or mismatched credential returns 401. The token comparison uses secrets.compare_digest. Note that POST /admin/index/rebuild is separately disabled on the results-server (allow_rebuild=False) and returns 503 regardless of the token — the index is rebuilt automatically at operator startup.
• Env var aliases. OperatorEnvironment parses these settings at process startup. AIPERF_OPERATOR_MUTATING_ROUTES_ENABLED and AIPERF_OPERATOR_MUTATING_ROUTES_TOKEN are the preferred, higher-precedence aliases; AIPERF_MUTATING_ROUTES_ENABLED and AIPERF_MUTATING_ROUTES_TOKEN remain compatibility aliases. routers/mutating_auth.py consumes only the validated typed settings.
• Helm configuration. The bundled chart does not template these variables — there is no resultsServer.mutatingRoutes value and no token-secret projection. Set AIPERF_OPERATOR_MUTATING_ROUTES_ENABLED and AIPERF_OPERATOR_MUTATING_ROUTES_TOKEN directly on the results-server container (e.g. via a deployment patch or a customized chart template).
• First-party callers. Clients must send the configured token as Authorization: Bearer <token> on protected POST requests. Note the rebuild route is mounted with allow_rebuild=False on the results-server and always returns 503; only a fresh bootstrap in the operator’s writer process rebuilds the index, so restart the operator pod instead. The browser dashboard does expose create/cancel controls: it prompts for the bearer token, holds it in sessionStorage for the tab’s lifetime, and sends it on those POSTs (ui/lib/api.js mutatingFetch, ui/components/token-modal.js). A 401 clears the stored token and re-prompts. aiperf kube and kubectl remain the token-free paths, since they talk to the apiserver directly rather than through the results-server.
• In-cluster RBAC. The sidecar uses its ServiceAccount token to call the Kubernetes API. Every /api/v1/jobs/* and /api/v1/cluster call runs with those permissions, so list aiperfjobs, get pods, patch aiperfjobs/spec, and list nodes must be granted in the operator’s ClusterRole. RBAC failures surface as 401 / 403 propagated from kubernetes_asyncio.
• Network isolation. The Service is typically ClusterIP only. External access is expected to come via kubectl port-forward (trusted user), aiperf kube dashboard (trusted user), or an ingress controller. Add a NetworkPolicy if your cluster requires stricter pod-to-pod controls.
• Path traversal. File-serving endpoints resolve every {namespace}/{job_id}/{filename} under the results directory and reject resolved paths that escape the base (404). Callers cannot read files outside the PVC.

Do not expose the results server directly to the public internet without a proxy that enforces authentication in front of the read-only routes too.