Adaptive Search on Kubernetes

Adaptive search lets the cluster choose its own sweep points instead of exhausting a grid. The same BayesianSearchPlanner (Optuna-backed, with the Gaussian-process path supplied by BoTorch via the [botorch] extra) used in-process by aiperf profile --search-* runs cluster-side when an AIPerfSweep CR sets spec.sweep.type: adaptive_search. The planner proposes one variation at a time; the sweep-controller pod creates a child AIPerfJob per trial of that variation (one when multiRun.numRuns is unset or 1), waits for them to terminate, scores the objective, and asks for the next point. Convergence detection (max iterations, improvement-patience plateau, or coefficient-of-variation plateau) terminates the loop early when further evaluations stop helping.

Reach for adaptive search when the search space is too large to grid enumerate (e.g. concurrency 1–1000), when a single scalar objective captures what you care about, and when you want the proposed points materialized as ordinary AIPerfJobs so each iteration is durable, cancellable, and visible through normal kubectl get aiperfjob workflows. For the algorithm details, flag grammar, and search_history.json schema, defer to Bayesian-Optimization Outer Loop; for the in-process tutorial, see Adaptive Search.

Architecture

The kopf operator stays unaware of Bayesian Optimization: it only sees ordinary AIPerfJob create/delete events. Planner state — the Gaussian process model, trial history, and convergence accumulators — lives in the controller process. After a restart, the controller constructs a fresh planner and deterministically replays terminal child results through the canonical ask() / tell() loop before proposing a new point. search_history.json remains an output record rather than an input checkpoint; replay rebuilds it from the Kubernetes-owned children.

If sweep.randomSeed is omitted, the Kubernetes plan builder derives a stable planner seed from the immutable AIPerfSweep metadata.uid. An explicit randomSeed always wins. Every child also carries aiperf.nvidia.com/run-identity, a SHA-256 hash of its exact generated AIPerfJob.spec. A deterministic child name is reused only when both ownership and this execution contract match. A missing or mismatched identity fails the sweep instead of feeding metrics from a different configuration into planner history.

The optimization stack is pulled in through the AIPerf [botorch] extra (alias [optuna]) and is present on the controller-pod image; operator pods do not need it.

Minimal AIPerfSweep CR

A single-dimension search over phases.profiling.concurrency. Dimension paths are rooted inside the benchmark: block, so the benchmark. prefix is redundant and rejected by the validator. This example optimizes output token throughput on Llama 3.1 8B Instruct served by vLLM:

1
apiVersion: aiperf.nvidia.com/v1alpha1
2
kind: AIPerfSweep
3
metadata:
4
  name: bo-concurrency-llama8b
5
  namespace: bench
6
spec:
7
  benchmark:
8
    models: [meta-llama/Llama-3.1-8B-Instruct]
9
    endpoint:
10
      urls: [http://vllm.bench.svc.cluster.local:8000/v1/chat/completions]
11
      type: chat
12
      streaming: true
13
    datasets:
14
      - name: main
15
        type: synthetic
16
    phases:
17
      - name: profiling
18
        type: poisson
19
        rate: 1.0  # fixed; concurrency is what searchSpace below overrides
20
        duration: 120
21
  sweep:
22
    type: adaptive_search
23
    planner: bayesian
24
    searchSpace:
25
      - path: phases.profiling.concurrency
26
        lo: 1
27
        hi: 1000
28
        kind: int
29
    objectives:
30
      - metric: output_token_throughput
31
        stat: avg
32
        direction: maximize
33
    maxIterations: 30
34
    nInitialPoints: 5
35
    improvementPatience: 8
36
    plateauWindow: 5
37
    plateauThreshold: 0.01
38
    randomSeed: 42
39
  multiRun:
40
    numRuns: 3
41
    cooldownSeconds: 30

numRuns: 3 runs three benchmarks — three child AIPerfJobs, one per trial — for each proposed point and feeds their pooled objective back to the planner (objectivePooling, default mean). Confidence per point at the cost of triple the wall clock. Drop to numRuns: 1 for fastest iteration and exactly one AIPerfJob per proposed point.

Multi-dimensional search

Search over concurrency and Poisson rate jointly:

1
spec:
2
  sweep:
3
    type: adaptive_search
4
    planner: bayesian
5
    searchSpace:
6
      - path: phases.profiling.concurrency
7
        lo: 1
8
        hi: 500
9
        kind: int
10
      - path: phases.profiling.rate
11
        lo: 1.0
12
        hi: 50.0
13
        kind: real
14
    objectives:
15
      - metric: output_token_throughput
16
        stat: avg
17
        direction: maximize
18
    maxIterations: 40
19
    nInitialPoints: 8
20
  multiRun:
21
    numRuns: 2

kind: int declares an integer dimension — Optuna suggests integer parameters natively, no rounding step — while kind: real keeps floats. nInitialPoints (default 5) becomes the sampler’s n_startup_trials: that many randomly-drawn startup points are evaluated before the surrogate model takes over. Bump it for higher-dimensional spaces (rule of thumb: >= 2 * len(searchSpace)).

Status fields you can watch

The CRD declares typed counters in status:

status is a preserve-unknown object. runStates, currentChildRef, currentCell, and aggregation are declared there as open objects — the keys exist in the CRD but their inner shape is unvalidated.

Both totalVariations and maxTotalRuns are upper bounds — early plateau or improvement-patience convergence shrinks the actual count (the controller rewrites totalVariations to the number of distinct variation indexes it actually ran). This mirrors how the trial-level convergence rule (multiRun.convergence) can stop a grid sweep’s cell short of multiRun.numRuns.

$
$ kubectl -n bench get aiperfsweep bo-concurrency-llama8b
$
NAME                     PHASE       COMPLETED   TOTAL   FAILED   CURRENT   AGE
$
bo-concurrency-llama8b   Succeeded   54          90      0                  12m

The full BO trajectory — every proposed point, the per-iteration objective, the running best, and the convergence reason — lives in search_history.json, not on the CR. See Output layout below and the search_history.json schema.

A compact projection of that artifact is served on the sweep detail route as search_summary, so the dashboard and any API client can read the planner’s own verdict without downloading the trajectory:

$
$ curl -s http://aiperf-operator.aiperf-system:8081/api/v1/sweeps/bench/bo-concurrency-llama8b \
>
    | jq '.search_summary | {convergence_reason, stop_kind, iteration_count,
>
                             sla_filter_count, best_trials, boundary_summary}'

search_summary is null for grid-family sweeps, for adaptive sweeps whose trajectory has not been harvested to the operator PVC yet, and whenever the artifact is unreadable — a missing verdict degrades the response, it never fails it. The complete search_history.json remains downloadable from the per-epoch sweep artifacts routes.

Mutual exclusion rules

In the flat spec envelope, adaptive_search is one of the sweep.type discriminator values (alongside grid, zip, scenarios, sobol, and latin_hypercube), so “adaptive plus grid” is not even expressible — the discriminator picks one. The remaining gates protect the cardinality contract:

These prevent sweeping on top of sweeping and keep a single source of truth for variation generation.

Operator-managed gate exception

The operator sets AIPERF_OPERATOR_MANAGED=1 in every controller and worker pod, and cli_runner._reject_in_process_sweep_under_operator hard-fails any in-process magic-list sweep under that flag — so the cluster never sweeps on top of a sweep. Adaptive search is the exception: the controller pod is the BO driver, and each per-iteration child AIPerfJob sees a single-config plan (no is_sweep shape), so the gate never fires for adaptive runs. The exemption is documented in the _reject_in_process_sweep_under_operator docstring (src/aiperf/cli_runner/_multi_run.py).

Cancellation behaviour

Patching the parent AIPerfSweep with spec.cancel: true is cooperative end to end:

1. A background poller in the sweep-controller pod (_poll_cancel_flag in sweep_controller/main.py) re-reads the parent CR and flips an in-process cancel flag when spec.cancel is true.
2. The adaptive loop and the child-wait loop both consult that flag as a cancel_check callable at their await boundaries.
3. The currently-running child AIPerfJob is patched with spec.cancel: true (K8sChildJobExecutor._patch_child_cancel), drains to a terminal phase, and contributes its results.
4. The controller skips remaining iterations, runs aggregation over the children that did complete, and flushes search_history.json with convergence_reason: "cancelled".

search_history.json is rewritten after every iteration. Cancellation keeps the completed prefix immediately; after a controller restart, the same prefix is reconstructed by reusing terminal children and replaying their metrics into the restart-stable planner before execution advances.

Output layout

Artifacts land under the operator’s AIPERF_RESULTS_DIR PVC (default /data), scoped by namespace, sweep name, and the sweep run-epoch:

<AIPERF_RESULTS_DIR>/
  <namespace>/
    sweeps/
      <sweep-name>/
        <sweep-run-epoch>/
          search_history.json          # BO trajectory + convergence_reason
          aggregate.json               # durable parent aggregate
          children.json                # child manifest
          sweep_aggregate/             # mode-agnostic per-combination aggregate
            profile_export_aiperf_sweep.json
            profile_export_aiperf_sweep.csv
            manifest.json              # epoch lineage of every child run
    <sweep-name>-v00-t0/               # per-iteration child AIPerfJob
      <child-run-epoch>/
        profile_export_aiperf.json     # full per-trial artifacts
        ...
    <sweep-name>-v00-t1/
    <sweep-name>-v01-t0/
    ...

The operator harvests the whole tree from the sweep-controller’s results sidecar before the JobSet is deleted (the controller pod’s /results is an emptyDir), re-rooting it onto the operator PVC.

Per-iteration child names follow the same <sweep>-v<NN>[-t<N>] budget as grid sweeps (build_child_name in sweep_controller/_naming.py): the variation index is the BO iteration index (-v00 is the first proposed point), and the trial suffix is present whenever multiRun.numRuns > 1 or multiRun.convergence is set. Each child sits at the same path layer as a standalone AIPerfJob and is reachable through the standard /api/v1/results/<ns>/<sweep>-v00-t1/ endpoints.

sweep_aggregate/ carries the same per-combination CSV/JSON schema produced by grid sweeps — aggregate_sweep_and_export groups by stamped variation_values and is mode-agnostic, so downstream readers do not need to know the sweep was adaptive.

Where to read more

• Bayesian-Optimization Outer Loop — algorithm, flag grammar, search_history.json schema, convergence reasons.
• Adaptive Search tutorial — in-process walkthrough with aiperf profile --search-*.
• Parameter Sweeps and Multi-Run on Kubernetes — grid sweeps, multi-run confidence, cancellation, failure policy.
• search_history.json schema — exact JSON shape consumed by post-run tooling.
• AIPerfSweep CRD validation rules — full catalog of admission-time invariants.