Checking Your Customization Job Metrics

After completing a customization job, you can monitor its performance through training and validation metrics. You can access these metrics in three ways:

1. Using the API
2. Through MLflow (optional)
3. Using Weights & Biases (optional)

Prerequisites

$
# Set the base URL for NeMo Platform
$
export NMP_BASE_URL="http://localhost:8080" # Or your deployed platform URL
$
$
# Optional: Weights & Biases for experiment tracking
$
export WANDB_API_KEY="<your-wandb-api-key>"

1
import os
2
from nemo_platform import NeMoPlatform
3
4
client = NeMoPlatform(
5
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
6
    workspace="default",
7
)

Tutorial-Specific Prerequisites

• Completed customization job with a valid job name
• (Optional) A job created with spec.integrations.mlflow and access to its configured MLflow tracking server

Available Metrics

A customization job tracks every numeric metric its backend reports. The two you will always find are:

• Training Loss (train_loss): recorded every step the training framework logs one
• Validation Loss (val_loss): recorded on every validation pass

Alongside those you will typically see train_lr (learning rate) and train_grad_norm, plus whatever else the algorithm produces — a DPO job also reports train_preference_loss and val_accuracy, for example.

An algorithm that scores validation on something other than a loss reports no val_loss, and the series stays empty rather than filling with zeros.

How Metrics Are Named

Each metric is named <phase>_<metric>, where the phase is train or val and the metric keeps whatever name the training framework gave it. A metric reported during both training and validation therefore stays separate: DPO’s accuracy becomes train_accuracy and val_accuracy rather than one interleaved series.

train_loss and val_loss are simply what this rule produces for a metric named loss.

Where Metrics Appear

Each metric shows up in two places in a training task’s status_details:

• The latest value, as a top-level field under its full name (train_loss, train_lr, …). Present only when the metric was actually reported, so a missing field means no value rather than a zero.
• The full history, under metrics, as a list of {step, epoch, value} points per metric. This is what the loss curves in the UI are drawn from.

Non-numeric values a framework emits alongside the scalars — histograms, tables, nested dictionaries — are not charted and do not appear in either place.

How Often Metrics Are Sent

Recording and sending are separate, and only the second is throttled.

Every value your training framework logs is recorded, at full resolution. What is limited is how often the accumulated set is sent to the platform, because each update carries the whole history and is written more than once server-side. Reports are buffered in memory and sent at most once every ten seconds by default; a buffered report loses nothing, since its points travel with the next one that goes. The final values are always sent when the job finishes.

The practical effect is that a chart may lag the run by a few seconds, and the curve you eventually read is complete.

Controlling Metric Detail

Two optional fields under schedule.progress_reporting change this. Most jobs need neither.

• min_report_interval_seconds (default 10): the least time between updates reaching the platform. Raise it to spend less training time on reporting; lower it — or set 0, which sends every logged step — for a progress bar that moves more often. Recording is unaffected either way.
• time_series_metrics: which metrics keep a history rather than only a latest value. Names are the full ones you read back, so they carry the phase prefix, and glob patterns are accepted: ["*_loss", "*_lr"]. Omit the field to take your backend’s default, which keeps the loss, learning rate and gradient norm. Use ["*"] to keep a history of everything the backend reports.

A metric left out of time_series_metrics is still reported as a latest value on every update — only its history is dropped. That keeps throughput counters like train_tps costing one number instead of several hundred points.

Set them in the schedule block when you create the job:

1
schedule={
2
    "epochs": 3,
3
    "progress_reporting": {
4
        "min_report_interval_seconds": 30,
5
        "time_series_metrics": ["*_loss", "*_lr"],
6
    },
7
}

Viewing Your Metrics

Using the API

Get job status and training metrics through the platform Jobs service:

1
import os
2
from nemo_platform import NeMoPlatform
3
4
client = NeMoPlatform(
5
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
6
    workspace="default",
7
)
8
9
# Get job status with metrics
10
job_name = "my-sft-job"
11
status = client.jobs.get_status(name=job_name, workspace="default")
12
13
print(f"Job: {status.name}")
14
print(f"Status: {status.status}")
15
16
# Check training step progress
17
for step in status.steps or []:
18
    if step.name == "training":
19
        for task in step.tasks or []:
20
            details = task.status_details or {}
21
            print(f"Training Phase: {details.get('phase')}")
22
            print(f"Step: {details.get('step')}/{details.get('max_steps')}")
23
            print(f"Epoch: {details.get('epoch')}/{details.get('num_epochs')}")
24
            print(f"Training Loss: {details.get('train_loss')}")
25
            print(f"Validation Loss: {details.get('val_loss')}")
26
            print(f"Learning Rate: {details.get('train_lr')}")
27
            print(f"Gradient Norm: {details.get('train_grad_norm')}")

To read the curves rather than the latest values, use the metrics payload. It carries every metric the job reported, so iterating it picks up backend-specific ones without naming them in advance:

1
for name, points in (details.get("metrics") or {}).items():
2
    if not points:
3
        continue
4
    print(f"{name}: {len(points)} points, latest {points[-1]['value']}")

Using MLflow

If your customization job was created with an integrations.mlflow configuration (see MLflow Integration):

1. Access the MLflow UI at the configured tracking_uri
2. Locate the configured experiment_name (defaults to the output model name)
3. Find the configured run name (defaults to the customization job ID)
4. View detailed metrics, including training and validation loss curves, under the “Metrics” tab

Using Weights & Biases

If your customization job was created with W&B integration enabled (see Weights & Biases Integration):

1. Go to wandb.ai and navigate to your project
2. Find the run corresponding to your customization job
3. View training and validation loss curves, learning rate schedules, and other metrics under the run’s dashboard

1
from nemo_automodel_plugin.schema import AutomodelJobInput
2
3
# Create an Automodel job with W&B integration
4
spec = AutomodelJobInput(
5
    model="default/llama-3-2-1b",
6
    dataset={"training": "default/my-dataset"},
7
    training={"training_type": "sft", "finetuning_type": "lora"},
8
    schedule={"epochs": 3},
9
    batch={"global_batch_size": 16, "micro_batch_size": 1},
10
    optimizer={"learning_rate": 1e-4},
11
    integrations={
12
        "wandb": {
13
            "project": "my-finetuning-project",
14
            "entity": "my-team",
15
            "tags": ["fine-tuning", "llama"],
16
            "api_key_secret": "my-wandb-key",
17
        }
18
    },
19
)
20
21
job = client.customization.automodel.jobs.create(
22
    name="my-wandb-job",
23
    workspace="default",
24
    spec=spec,
25
)
26
27
print(f"Submitted job: {job.job.name}")

The api_key_secret field references a stored secret containing your WANDB_API_KEY. Use the secret name (e.g., "my-wandb-key") to resolve it from the request workspace. To create the secret, see Weights & Biases Keys.

Then view your results at wandb.ai under your project.