Workflow

Workflow manages a single certification category run. It is created by the Certification controller — one per category — and is not typically created directly by users.

Spec fields

Fields documented so far:

Because spec.jobTemplate.spec is a plain JobSpec, its workloadMetadata cannot be added, removed, or changed after the Workflow is created. Editing it mid-flight is rejected, so every group and iteration produces Jobs with the same workload labels. Raw spec.overrides remain editable; a Job’s own metadata becomes immutable at the moment that Job is created.

The measurement fields in spec.jobTemplate.spec follow the same transition rules: nodeHealthMonitor, goodputMeasurement, and bandwidthMeasurement cannot be added, removed, or changed after the Workflow is created. Other mutable fields in the Job template remain editable.

Gang scheduling consistency contract

spec.gangScheduler is not a second place to configure gang scheduling — it is a consistency contract. When a WorkloadRun or Certification configures gang scheduling, the generated Workflow records the intent with its queue and queue label key already resolved to their effective values.

The field sits outside jobTemplate, dependencies, and orchestration, so no override can reach it, and it is immutable in both presence and value. That is what lets a rendered Workflow submitted on its own enforce the same contract without fetching its original owner, and without relying on mutable annotations.

After every matching override resolves, and before dependencies or Jobs are created, the Workflow controller checks the effective manifests against the intent:

• The workload object’s queue label is NVCRE’s to write, so an override that removed it gets it restored on the generated Job, and an override that changed it fails.
• Every effective runtime Job-template and pod-template queue label must equal the configured queue, and pod scheduler names must match the configured scheduler. TrainJob runtimePatches are folded in when computing the effective values, for launcher and worker templates alike. A missing or conflicting runtime value fails and is never silently repaired — rewriting the operator’s own runtime override would hide the disagreement rather than surface it.
• An override that repoints runtimeRef at a runtime outside the Workflow’s dependencies fails, as does a runtime shape that prevents checking these fields at all.

The check runs again before each later group or iteration is launched, so no subsequent override or runtime-patch mutation can bypass it. A conflict marks the Workflow Failed with reason GangSchedulingConflict before any child is created. The restore half writes only to the working copy used to create children; the stored Workflow’s immutable field is never patched.

Workflows without gangScheduler — including hand-authored ones — receive workload-label validation and passthrough, not inferred queue checks, and keep responsibility for their own scheduling configuration.

Accepted and conflicting overrides

Take a Workflow whose persisted intent is queue team-a:

spec:
  gangScheduler:
    schedulerName: kai-scheduler
    queue: team-a
    queueLabelKey: kai.scheduler/queue
  jobTemplate:
    spec:
      workloadMetadata:
        labels:
          kai.scheduler/queue: team-a
      workload:
        trainJob:
          runtimeRef:
            kind: TrainingRuntime
            name: nccl-runtime

An override that touches unrelated fields, or that removes the workload object’s queue label, is accepted. The removed label is restored on the generated Job:

spec:
  overrides:
    - when:
        platform:
          equals: aws
      jobTemplatePatch:
        - op: remove
          path: /spec/workloadMetadata/labels/kai.scheduler~1queue

An override that changes the workload object’s queue label fails the Workflow with reason GangSchedulingConflict, naming the key, both values, and the field path:

spec:
  overrides:
    - when:
        platform:
          equals: aws
      jobTemplate:
        spec:
          workloadMetadata:
            labels:
              kai.scheduler/queue: team-b

spec.jobTemplate.spec.workloadMetadata.labels["kai.scheduler/queue"]: conflicting label value "team-b", expected "team-a"

An override that rewrites a runtime queue label fails the same way and the runtime is left as the override wrote it, never repaired:

spec:
  overrides:
    - when:
        platform:
          equals: aws
      dependencies:
        - apiVersion: trainer.kubeflow.org/v1alpha1
          kind: TrainingRuntime
          metadata:
            name: nccl-runtime
          spec:
            template:
              spec:
                replicatedJobs:
                  - name: node
                    template:
                      metadata:
                        labels:
                          kai.scheduler/queue: team-b

gangScheduler is configured with queue "team-a" but TrainingRuntime "nccl-runtime" replicatedJob "node" sets label "kai.scheduler/queue" to "team-b" on template.metadata.labels

Status fields

Placement

spec.orchestration.placement decides how many jobs a Workflow runs and who picks the nodes.

Pinned is the default and the behavior every Workflow has always had. The discovered target is partitioned into groups of nodesPerJob, every node lands in some group, and each job gets a required kubernetes.io/hostname node affinity naming its group’s machines. A 12-node target with nodesPerJob: 2 produces six jobs.

Unpinned runs exactly one job of exactly nodesPerJob nodes, no matter how many the target matches, and carries no hostname affinity. The same 12-node target with nodesPerJob: 2 produces one two-node job and leaves the other ten untouched. The scheduler chooses the machines, constrained to the target by node affinity.

The requested size is honored exactly under Unpinned, or the run fails saying why. It is never clamped to fit the fleet, snapped to the nearest size the model constraints allow, or auto-selected when no size is given.

Both modes carry target.nodeSelector, target.matchExpressions, and the detected GPU products onto every pod as a required node affinity, so the target is enforced by the scheduler at bind time rather than only at discovery time.

Two fields are incompatible with Unpinned and fail the Workflow with reason PartitionError:

• orchestration.diagnose, which is definitionally about subdividing a group to isolate a fault.
• orchestration.topology.strictDomain, which asks for one job per topology domain.

orchestration.topology.topologyKey on its own is ignored, not rejected. It only feeds the partitioner, which Unpinned does not run, and it is injected by catalog overrides on GB200 and GB300 rather than written by the operator. Rejecting it would fail those runs with a message about a field that does not appear in the submitted YAML. To confine an unpinned job to one topology domain, name the domain label in target.matchExpressions, which is carried onto the pods.

Naming

Workflows are named <certificationName>-<domain>-<variant>.

Lifecycle

1. Pulls WorkflowSpec from the catalog for the assigned {domain, variant}.
2. Detects platform and GPU architecture; applies matching overrides.
3. Creates a child Job.
4. Manages iteration count; marks Succeeded or Failed when complete.