API Reference

BandwidthMeasurement

View as Markdown

BandwidthMeasurement watches a Job’s NCCL log output and computes per-message-size bus bandwidth metrics for collective operations. It is automatically created by the Job controller when spec.bandwidthMeasurement is configured, or can be created manually.

Spec fields

FieldTypeDescription
jobRefTypedLocalObjectReferenceThe Job whose pod logs to watch
logProfileRefstring (required)Name of the cluster-scoped LogProfile that defines the bandwidthResult regex pattern
sampleIntervalDurationHow often to sample pod logs while the Job is running. Default: 60s
testTypestringNCCL collective operation identifier (e.g., all_reduce, alltoall). Used as the nccl_test Prometheus label

Status fields

FieldTypeDescription
results[]BandwidthResultPer-message-size average bandwidth measurements
startTimeTimeWhen measurement started (when the referenced Job began running)
completionTimeTimeWhen measurement completed (when the referenced Job reached a terminal state)
conditions[]ConditionCurrent state: Measuring (in progress) or Complete (finished)

Each BandwidthResult entry contains:

FieldTypeDescription
sizeBytesint64Message size in bytes
algBWstringAverage algorithmic bandwidth in GB/s
busBWstringAverage bus bandwidth in GB/s
samplesintNumber of measurements averaged for this size

How it works

  1. Watches the pod logs of the referenced Job at sampleInterval.
  2. Applies the bandwidthResult regex pattern from the referenced LogProfile to extract size, algBW, and busBW capture groups.
  3. Computes a running average per message size across all observed measurements.
  4. Sets Complete when the Job reaches a terminal state.