Skip to navigation
API Reference

GoodputMeasurement

View as Markdown

GoodputMeasurement watches a Job’s pod logs in real time, applies regex patterns from a LogProfile, and computes a goodput ratio — the fraction of elapsed training time the job was making useful forward progress. It is automatically created by the Job controller when spec.goodputMeasurement is configured, or can be created manually.

Spec fields

FieldTypeDescription
jobRefTypedLocalObjectReferenceThe Job whose pod logs to watch
logProfileRefstring (required)Name of the cluster-scoped LogProfile defining parsing patterns
sampleIntervalDurationHow often to sample pod logs while the Job is running. Default: 60s

Status fields

FieldTypeDescription
resultstringCurrent runtime goodput ratio (0.0–1.0) as a string
startTimeTimeWhen measurement started
completionTimeTimeWhen measurement completed
conditions[]ConditionCurrent state: Measuring (in progress) or Complete (finished)

Additional diagnostic fields (all optional):

FieldTypeDescription
currentStepintLatest training step observed from logs
highestStepintHighest training step ever reached (may exceed currentStep after restart)
trainingTimeSecstringTotal training wall-clock time in seconds
lostWorkTimeSecstringCumulative lost work time across all interruptions
interruptionCountintNumber of interruptions detected
avgStepTimeSecstringAverage time per training step (excluding warmup)
avgTFLOPSPerGPUstringAverage TFLOPS per GPU from training step logs

How it works

  1. Watches pod logs via the PodLogFetcher interface at sampleInterval.
  2. Applies named capture group regexes from the referenced LogProfile.
  3. Computes the goodput ratio: (t_w - t_ch - t_rm - t_re - t_save) / (t_w - t_re) where t_w = training time, t_ch = lost work time, t_rm = resume time, t_re = reschedule time, t_save = checkpoint save time.
  4. Sets Complete when the referenced Job reaches a terminal state.

Threshold evaluation (pass/fail) is done at the Job tier, not here — see Job.spec.thresholds.

A measurement the Job controller creates records the Job’s UID in the nvcre.nvidia.com/job-uid annotation. The Workflow owns the measurement, so it outlives its Job. A retried Workflow group gives each attempt its own Job name, and so its own measurement, and the earlier attempt’s measurement is deleted when that attempt is retired. A Job can still be replaced under the same name, for example a retry relaunched by a controller version from before attempt-indexed names. A measurement left by such an earlier Job is not used for the new Job’s thresholds or stall detection: the Job controller deletes it and creates a fresh one for the new Job, the same way it treats a BandwidthMeasurement. A measurement without the annotation was created before it existed and is matched by Job name.