API Reference

LogProfile

View as Markdown

LogProfile is a cluster-scoped resource that defines regex patterns with named capture groups for parsing training framework log output. It is referenced by GoodputMeasurement and BandwidthMeasurement resources.

Example

1apiVersion: cre.nvidia.com/v1alpha1
2kind: LogProfile
3metadata:
4  name: my-framework
5spec:
6  timestamp:
7    layout: "2006-01-02T15:04:05.999999999Z"
8  patterns:
9    trainingStep:
10      regex: 'reduced_train_loss=(?P<loss>[0-9.]+).*step_time=(?P<stepTiming>[0-9.]+)'
11      example: 'reduced_train_loss=0.1234 step_time=2.5'
12      units:
13        stepTiming: s
14    applicationStart:
15      regex: 'Training started'

Spec fields

spec.timestamp (required)

Configures how to parse (?P<timestamp>...) named captures from log lines.

FieldTypeDescription
layoutstring (required)Go time layout string for parsing captured timestamps. Example: "2006-01-02T15:04:05.999999999Z"

spec.patterns

A LogPatternSet — a named struct (not a list) with 8 optional fields, each an EventPattern. Each pattern uses Go named capture groups (?P<name>...).

FieldCapturesDescription
trainingSteptimestamp, globalStep, iteration, epoch, stepTiming, loss, tflops, elapsedTimeTraining iteration log lines
checkpointSavetimestamp, step, path, saveDurationCheckpoint save start
checkpointDonetimestamp, step, pathCheckpoint save completion
checkpointRestoretimestamp, path, stepCheckpoint load start
checkpointLoadedtimestamp, path, stepCheckpoint load completed
applicationStarttimestampApplication framework start marker
warmupStep—Warmup/startup iteration marker (presence of match is sufficient)
bandwidthResultsize, algBW, busBWNCCL bandwidth test result lines (used by BandwidthMeasurement)

Each EventPattern has:

FieldTypeDescription
regexstring (required)Go regex with named capture groups
examplestringSample log line the regex should match (documentation + validation)
unitsmap[string]stringUnits for duration captures: "s", "ms", "us". Only needed for stepTiming, elapsedTime, saveDuration. Defaults to "s".

Additional spec fields

FieldTypeDescription
workerStrategyWorkerStrategySpecHow to read logs from multi-worker jobs (Single or Multi). Default: Single (worker-0 only).
containerNamestringContainer to read logs from. Auto-detected if empty.
warmupStepsintNumber of initial steps per run to flag as warmup and exclude from goodput calculations.
logIntervalintTraining log interval (e.g., --log-interval 10). Filters intermediate steps.

Built-in profiles

The following LogProfiles are installed by the Helm chart:

NameFramework
megatron-trainingMegatron-LM training logs
megatron-bridgeMegatron bridge logs
nccl-bandwidthNCCL test output (bandwidth results)
nccl-loopbackNCCL loopback test output

Scope

LogProfiles are cluster-scoped — a single profile can be referenced by Jobs across all namespaces.