API Reference

Job

View as Markdown

Job creates and monitors the actual workload (a TrainJob or other adapter-supported resource). It is created by the Workflow controller and is not typically created directly by users.

Spec fields

Generated from CRD schema — coming soon.

Status fields

FieldTypeDescription
conditions[]ConditionExclusive set: InProgress, Succeeded, Failed. Independent (additive): HardwareFailed (can be True alongside execution state), ValidationFailed (can be True alongside Succeeded)
workloadRefWorkloadReferenceReference to the created workload (TrainJob)
failedNodes[]FailedNodeNodes identified as failed; each entry has name, reason, and optional message
restartCountint32Number of checkpoint-based restarts
failureLogFailureLogTail of pod logs from the most recent failure (pod name, node, exit code, log tail)

Each FailedNode entry has:

FieldTypeDescription
namestringKubernetes node name
reasonstringHardwareFailureDetected, ThresholdViolation, or WorkloadFailed
messagestringDetailed failure message

GoodputMeasurement and BandwidthMeasurement resources reference the Job via their own spec.jobRef — the Job does not hold references to them.

Naming

Jobs are named <workflowName>-job.

Lifecycle

  1. Creates the workload via the adapter pattern (selects adapter from WorkloadSpec).
  2. Runs NodeFailureDetector concurrently.
  3. Optionally creates GoodputMeasurement or BandwidthMeasurement.
  4. On workload completion, evaluates health results and performance thresholds.
  5. Marks Succeeded or Failed.