> This page is for NVIDIA Cluster Readiness Engine, version v0.8.0.
> For other versions, use one of these documentation indexes:
> - Latest · v0.8.2 (default): https://docs.nvidia.com/cluster-readiness-engine/v0.8.2/llms.txt
> - v0.8.1: https://docs.nvidia.com/cluster-readiness-engine/v0.8.1/llms.txt
> - v0.8.0: https://docs.nvidia.com/cluster-readiness-engine/v0.8.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/cluster-readiness-engine/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/cluster-readiness-engine/_mcp/server.

# Job

> CRD reference for the Job resource.

`Job` creates and monitors the actual workload (a `TrainJob` or other adapter-supported resource). It is created by the `Workflow` controller and is not typically created directly by users.

## Spec fields

| Field | Type | Description |
|-------|------|-------------|
| `workload` | WorkloadSpec | Required, immutable. Discriminated union selecting the workload framework; exactly one field must be set |
| `workloadMetadata` | WorkloadMetadata | Optional, immutable. Labels applied to the generated workload object itself (see below) |
| `nodeHealthMonitor` | NodeHealthMonitor | Optional, immutable in presence and value. Configures node-health monitoring |
| `goodputMeasurement` | GoodputMeasurementConfig | Optional, immutable in presence and value. Configures goodput measurement |
| `bandwidthMeasurement` | BandwidthMeasurementConfig | Optional, immutable in presence and value. Configures bandwidth measurement |
| `workloadMetadata.labels` | map[string]string | Optional. At most 32 entries. Label keys must be valid Kubernetes label keys and values valid Kubernetes label values; `app.kubernetes.io/managed-by` and any key under `nvcre.nvidia.com/` are rejected |

### Workload object labels

`spec.workloadMetadata.labels` is the canonical place NVCRE records labels for the **workload object** it creates from `spec.workload` — today a Kubeflow `TrainJob`. `WorkloadRun` and `Certification` both resolve their own `workloadMetadata` into this field, and a hand-authored `Workflow` writes `spec.jobTemplate.spec.workloadMetadata` directly.

<Warning>
These labels are executable policy, not decoration. They are how a workload reaches a Kueue local queue, a KAI Scheduler queue, or any other integration keyed on the submitted object's labels — all of which are read **before** workload pods exist. A label here can cause an external system to suspend, mutate, reject, prioritize, or bill the workload. Treat the field as part of the submission contract.
</Warning>

```yaml
apiVersion: nvcre.nvidia.com/v1alpha1
kind: Job
metadata:
  name: my-job
spec:
  workloadMetadata:
    labels:
      kueue.x-k8s.io/queue-name: gpu-team-a   # Kueue local queue
      kai.scheduler/queue: burn-in            # KAI Scheduler queue
      environment: burn-in                    # arbitrary label
  workload:
    trainJob:
      runtimeRef:
        kind: TrainingRuntime
        name: nccl-all-reduce-runtime
```

The resulting `TrainJob` carries those three labels plus the two the controller owns, `app.kubernetes.io/managed-by: nvcre` and `nvcre.nvidia.com/job: my-job`.

**This is not pod-label injection.** `workloadMetadata.labels` label the workload object and nothing below it: they are not copied to `JobSet`s, pod templates, or pods. Conversely, ordinary Job labels — `Workflow` `spec.jobTemplate.metadata.labels` — label the NVCRE `Job` only and are never copied here. Neither direction is inferred, because propagating labels automatically could opt a workload into an admission controller or a queue that nobody asked for. If you need a label at the pod level, set it in the workload spec's own pod templates.

**Reserved keys.** `app.kubernetes.io/managed-by` and any key under the `nvcre.nvidia.com/` prefix identify controller ownership and Job association. Setting either is rejected at admission rather than silently overwritten.

**Limits.** At most 32 labels. Keys follow the Kubernetes qualified-name grammar: an optional DNS-subdomain prefix of at most 253 characters, a `/`, then a name of at most 63 characters. Values follow Kubernetes label-value syntax and may be empty — an empty value is a real label, not a deletion. There is no deletion syntax.

**Immutability.** `workloadMetadata` cannot be added, removed, or changed after the Job is created; all three are rejected by CRD transition rules. A checkpoint restart therefore recreates the workload with the labels it was originally admitted with, so a restart cannot land the workload in a different queue than the one that admitted it. To change the labels, create a new Job (or a new `WorkloadRun`/`Certification`).

**Measurement configuration immutability.** `nodeHealthMonitor`, `goodputMeasurement`, and `bandwidthMeasurement` are each immutable in both presence and value. An optional field omitted when the Job is created cannot be added later, and a configured field cannot be removed or changed. Other mutable Job fields remain editable.

## Status fields

| Field | Type | Description |
|-------|------|-------------|
| `conditions` | []Condition | Exclusive set: `InProgress` (`WorkloadCreated`, `WorkloadPending`, `WorkloadRunning`, `WorkloadSchedulingBlocked`, `WorkloadCreationRejected`, `WorkloadRestarting`), `Succeeded`, `Failed` (`WorkloadFailed`, `WorkloadStalled`, `WorkloadCreationError`, plus Workflow-set `JobTimedOut`). Independent (additive): `HardwareFailed` (can be True alongside execution state), `ValidationFailed` (can be True alongside `Succeeded`) |
| `workloadRef` | WorkloadReference | Reference to the created workload (`TrainJob`) |
| `workloadStartTime` | Time | When the workload was first observed running rather than pending (e.g. suspended by Kueue). `timeoutPerJob` is measured from this timestamp, and the stall clock never starts before it, so queued time counts against neither. Cleared on checkpoint restart |
| `schedulingBlockedSince` | Time | When the workload's pods were first observed unschedulable (`PodScheduled=False/Unschedulable`) in the current blocked episode. Cleared when no pod is blocked. While set, the `timeoutPerJob` clock is paused at this instant and stall detection is skipped; past the grace window the Job reports `InProgress` with reason `WorkloadSchedulingBlocked`. The pause is bounded: an episode that lasts longer than `timeoutPerJob` times the Job out. On recovery `workloadStartTime` is advanced by the paused interval, so runtime consumed before the block still counts. See [ADR-083](../designs/083-scheduling-stall-visibility.md) |
| `schedulingResumedTime` | Time | When the most recent scheduling-blocked episode ended. Startup- and training-stall detection measure from this instant when it is later than their own anchor (application start, or the last observed step). See [ADR-083](../designs/083-scheduling-stall-visibility.md) |
| `failedNodes` | []FailedNode | Nodes identified as failed; each entry has `name`, `reason`, and optional `message` |
| `workloadCreationRejectedSince` | Time | When the API server first rejected the workload create (admission webhook or policy denial, ResourceQuota, validation). Cleared when the create succeeds. While set, the Job reports `InProgress` with reason `WorkloadCreationRejected`; a rejection that outlasts the workload creation deadline (default 10 minutes) fails the Job with `WorkloadCreationError`. A `Forbidden` create that a SelfSubjectAccessReview shows is an RBAC denial does not start this clock. See [#488](https://github.com/NVIDIA/cluster-readiness-engine/issues/488) |
| `restartCount` | int32 | Number of checkpoint-based restarts |
| `failureLog` | FailureLog | Tail of pod logs from the most recent failure (pod name, node, exit code, log tail) |

Each `FailedNode` entry has:

| Field | Type | Description |
|-------|------|-------------|
| `name` | string | Kubernetes node name |
| `reason` | string | `HardwareFailureDetected`, `ThresholdViolation`, or `WorkloadFailed` |
| `message` | string | Detailed failure message |

`GoodputMeasurement` and `BandwidthMeasurement` resources reference the Job via their own `spec.jobRef` — the Job does not hold references to them.
If the workload is admitted but its pods cannot be placed, the `InProgress` condition carries reason `WorkloadSchedulingBlocked` and the message relays the scheduler's own diagnosis from the pod's `PodScheduled` condition. Blocked time does not count against `timeoutPerJob` or stall detection, but a single blocked episode longer than `timeoutPerJob` times the Job out, so a Job whose pods can never schedule still terminates; the grace window before the reason surfaces is tunable via `spec.schedulingStallGraceSeconds` (default 5 minutes). See [ADR-083](../designs/083-scheduling-stall-visibility.md).

If the API server rejects the workload create (an admission webhook or policy denial, or an exhausted ResourceQuota), the controller keeps retrying with backoff, since some rejections clear on their own, and the `InProgress` condition carries reason `WorkloadCreationRejected` with the rejection message. The Workflow relays it on its `InProgress` condition. A rejection that lasts longer than the workload creation deadline (default 10 minutes, set with the manager's `--workload-creation-deadline` flag or Helm `manager.workloadCreationDeadline`, independent of `timeoutPerJob`) fails the Job with reason `WorkloadCreationError`. That failure leaves `status.failedNodes` empty, so the group's nodes do not reach the Certification's failed-node list: the rejection is about the object, not the hardware. The deadline is per Job attempt: with `spec.orchestration.execution.retryFailedGroups` set, the replacement Job starts with an empty status and its own clock, so a permanent denial costs (retries + 1) times the deadline before the group gives up. A `Forbidden` create is confirmed with a SelfSubjectAccessReview first; an RBAC denial (a missing RoleBinding) is not a rejection and keeps retrying without the deadline. Errors that do not reject the object, such as timeouts or throttling, keep retrying without a deadline.

## Naming

Jobs are named `<workflowName>-job`.

## Lifecycle

1. Creates the workload via the adapter pattern (selects adapter from `WorkloadSpec`).
2. Runs `NodeFailureDetector` concurrently.
3. Optionally creates `GoodputMeasurement` or `BandwidthMeasurement`.
4. On workload completion, evaluates health results and performance thresholds.
5. Marks Succeeded or Failed.