CRD Validation Rules

When you kubectl apply an AIPerfJob or AIPerfSweep CR, the Kubernetes apiserver runs two layers of validation before the operator ever sees the resource:

1. Structural schema — types, enums, minimum/maximum, required. If this layer rejects, the resource is never persisted.
2. CEL x-kubernetes-validations — cross-field invariants compiled into the CRD. These mirror the Pydantic @model_validator rules on the underlying config models (EndpointConfig, RuntimeConfig, MultiRunConfig) and on AIPerfJobSpec / AIPerfSweepSpec, but fire at admission time so a bad CR is rejected with a clear message before any pod is scheduled.

The CRD that defines both layers is auto-generated from the AIPerfJobSpec and AIPerfSweepSpec Pydantic models in src/aiperf/kubernetes/crd_models.py — see the dev flow doc for how to add new rules.

The lifecycle fields spec.cancel and spec.skipEndpointCheck are native booleans, and spec.ttlSecondsAfterFinished is a nonnegative integer. Quoted forms such as cancel: "false" or ttlSecondsAfterFinished: "300" are rejected at admission. Cancellation and retention handlers consume raw Kubernetes spec values, so native schema typing is part of the lifecycle safety boundary.

Shorthand acceptance

The whole structural required chain is spec: [benchmark], spec.benchmark: [endpoint], endpoint: [urls], and spec.benchmark carries an empty x-kubernetes-validations: [] block. models and every shorthand sibling (model, dataset, warmup, profiling) are typeless x-kubernetes-preserve-unknown-fields properties; datasets and phases are type: array with opaque preserve-unknown items (datasets additionally carries minItems: 1 / maxItems: 1 because the runtime loads exactly one dataset, phases carries minItems: 1). No CEL rule requires or excludes any of them: a typeless field cannot be has()-ed and opaque array items cannot be dereferenced. So the apiserver accepts either idiom, and the operator’s before-validator does the actual shorthand↔canonical normalization on reconcile. This means kubectl apply accepts the CLI-YAML idiom without rewriting:

1
# Shorthand form — accepted by the apiserver, normalized by the operator's
2
# before-validator on reconcile.
3
spec:
4
  benchmark:
5
    endpoint:
6
      urls: ["http://server:8000/v1/chat/completions"]
7
      type: chat
8
    model: meta-llama/Llama-3.1-8B-Instruct  # singular, scalar
9
    dataset:                                 # singular, dict
10
      type: synthetic
11
    profiling:                               # phase shorthand
12
      type: concurrency

1
# Canonical form — also accepted; same post-normalization shape, except that
2
# the `dataset:` shorthand above injects `name: default` rather than `main`.
3
spec:
4
  benchmark:
5
    endpoint:
6
      urls: ["http://server:8000/v1/chat/completions"]
7
      type: chat
8
    models: [meta-llama/Llama-3.1-8B-Instruct]
9
    datasets:
10
    - name: main
11
      type: synthetic
12
    phases:
13
    - name: profiling
14
      type: concurrency

One more node is deliberately typeless for the same reason: spec.benchmark.artifacts.userFiles[].content. A user file’s body is a bare string for format: text and a dict/list/scalar otherwise, so no single type: matches every legal value. The node carries x-kubernetes-preserve-unknown-fields: true with no type, and the format↔content pairing is enforced by the UserFile Pydantic validators on reconcile. path and content stay structurally required.

You cannot mix the two forms for the same slot — the operator’s normalize_before_validation raises a Pydantic ValueError on reconcile (status.phase=Failed with 'dataset' cannot be used with 'datasets'. Use 'dataset' for a single dataset or 'datasets' for multiple named datasets.). The check can’t move to CEL because the shorthand fields are typeless preserve-unknown siblings — see Operator-side (Pydantic) invariants below.

Rule catalog

The tables below list only the CEL rules tools/generate_crd.py actually emits (verified against the generated crd-aiperfjob.yaml / crd-aiperfsweep.yaml). Each entry gives the verbatim CEL expression and the message users see on rejection. Cross-field invariants that CEL cannot express are enforced by Pydantic on the operator side instead — see Operator-side (Pydantic) invariants below.

Endpoint rules — spec.benchmark.endpoint (both kinds)

The form-data endpoint list in the third rule is derived at generation time from the requires_form_data plugin metadata in src/aiperf/plugin/plugins.yaml, so it cannot drift from the Pydantic gate.

Runtime rules — spec.benchmark.runtime (both kinds)

MultiRun rules — spec.multiRun (both kinds)

No CEL rule is attached to spec.benchmark.artifacts on either kind.

AIPerfJob spec-level rules

AIPerfSweep spec-level rules

Workload update rules

Create-time workload fields use this presence-safe transition rule on the parent spec node:

Keeping the rule on spec, which exists on every update, is intentional. A transition rule attached to an optional field is not evaluated when that field is added or removed. The explicit has parity rejects a value change, first-set-after-create, and removal. Kubernetes evaluates rules using oldSelf only on updates, so initial creation remains unrestricted.

The exact update contract is:

The mutable whitelist follows the live reconciliation paths. The AIPerfJob monitor rereads timeoutSeconds; both kinds watch or poll cancel; and the operator’s parent-sweep reaper rereads ttlSecondsAfterFinished. Every other field is rendered into the ConfigMap, JobSet, serialized sweep plan, or child template during creation. Accepting updates to those fields would change the CR without changing the running workload and could make status.observedGeneration falsely acknowledge a configuration the pods never received.

The spec.sweep absence rule is AIPerfJob-only; the AIPerfSweep CRD instead asserts spec.sweep is present. spec.benchmark carries an empty x-kubernetes-validations: [] on both kinds.

Operator-side (Pydantic) invariants

Several cross-field invariants cannot be expressed in CEL — either the fields they reference are typeless preserve-unknown siblings (model, dataset, warmup, profiling), or the phases[] / datasets[] array items are emitted as opaque x-kubernetes-preserve-unknown-fields blobs (they are heterogeneous Pydantic discriminated unions). CEL has(self.X) won’t compile against a typeless field, and opaque array items cannot be dereferenced. These checks stay in the operator’s @model_validator decorators and surface only on reconcile (they are also run client-side by aiperf kube validate):

There is no validate_datasets_unique_names or validate_dataset_references validator — dataset-name presence is checked by parse_datasets_input and phase↔dataset coupling by validate_phase_dataset_compatibility.

If you submit a CR that the apiserver accepts but the operator later rejects, the failure shows up as status.phase=Failed with the validation error in status.error (or in operator pod logs).

Example error messages

Each CEL rejection names the rule that fired, so the failure points directly at what to fix.

$ kubectl apply -f bad-template.yaml
The AIPerfJob "x" is invalid: spec.benchmark.endpoint: Invalid value: "object":
  endpoint.template is required when endpoint.type='template'
$ kubectl apply -f sweep-as-job.yaml
The AIPerfJob "x" is invalid: spec: Invalid value: "object":
  AIPerfJob.spec.sweep must be null/omitted. Use kind: AIPerfSweep for
  parameter sweeps.
$ kubectl apply -f job-as-sweep.yaml
The AIPerfSweep "x" is invalid: spec: Invalid value: "object":
  AIPerfSweep.spec.sweep is required. Use kind: AIPerfJob for single
  benchmarks.

The shorthand↔canonical mutual-exclusion error is not a CEL rejection at kubectl apply; it surfaces on reconcile as status.phase=Failed with the Pydantic message 'dataset' cannot be used with 'datasets'. Use 'dataset' for a single dataset or 'datasets' for multiple named datasets.

Extending the rule set

Adding a new CEL rule is a small change in tools/generate_crd.py:

1. Decide which shape the rule applies to (benchmark, endpoint, runtime, multiRun) and pick the matching _decorate_*_node helper, or add a new shape detector if your target node has a unique property fingerprint.

2. Append a {"rule": ..., "message": ...} entry to that helper’s _add_validation_rules(...) call.

3. Add a structural assertion to tests/unit/operator/test_aiperfsweep_crd_generation.py.

   A shape detector keys off property names, so renaming a field silently retires every rule attached to its node — the detector stops matching and the generator reports nothing. Two decorators had been inert this way. When you rename or nest a spec field, re-read the detector that fingerprints it, and prefer a structural test that asserts the rule is present on both kinds over one that only asserts a rule’s text.

4. Regenerate (uv run python tools/generate_crd.py) and verify the regen is idempotent (tools/generate_crd.py --check).

5. Round-trip against a real apiserver (kind cluster + kubectl apply --dry-run=server) — the CEL compiler runs at CRD-install time and will reject rules that reference undeclared fields or opaque items.

CEL constraints that aren’t obvious from the Pydantic side:

• has(self.X) only works on properties that are declared in the schema. Properties under x-kubernetes-preserve-unknown-fields are invisible to CEL.
• Array items emitted as opaque preserve-unknown blobs cannot be dereferenced (no phases[].name, no phases[0].seamless).
• oldSelf is only available in transition rules and triggers on update. Use !has(oldSelf.X) || oldSelf.X == self.X for “first-set freezes” semantics.
• The K8s apiserver compiles CEL at CRD install time; rule errors fail the install with a clear compilation failed: undefined field 'X' message.

See also

• docs/dev/kubernetes-flow.md — operator/CR lifecycle, including how the CRD generator decorator pattern is wired.
• docs/kubernetes/validate.md — aiperf kube validate runs the same schema check client-side so CI catches violations before kubectl apply.
• docs/kubernetes/configuration.md — full CR-field reference.