Checking Your Customization Job Metrics#

After completing a customization job, you can monitor its performance through training and validation metrics. There are three ways to access these metrics:

  1. Using the API

  2. Through MLflow (optional)

  3. Using Weights & Biases (optional)

Note

The time to complete this tutorial is approximately 10 minutes.

Available Metrics#

Each customization job tracks two key metrics:

  • Training Loss: Calculated during training, logged every 10 steps

  • Validation Loss: Calculated during validation, logged every epoch

Prerequisites#

  • Completed customization job with a valid ID

  • (Optional) Access to NeMo with MLflow tracking enabled

  • (Optional) Weights & Biases account and API key for enhanced visualization

Viewing Your Metrics#

Using the API#

Get metrics with a simple API call:

curl ${CUST_HOSTNAME}/v1/customization/jobs/${customizationID}/status

The response includes timestamped training and validation loss values.

Using MLflow#

  1. Access the MLflow UI (typically available through your cluster’s external URL)

  2. Find your experiment using the customization ID

  3. Select the run to view metrics under “Metrics”

Using Weights & Biases#

To enable W&B integration, include your API key when creating a customization job in the call header:

curl --location "https://${CUST_HOSTNAME}/v1/customization/jobs" \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'wandb-api-key: <WANDB_API_KEY>' \
--data '{
    "config": "meta/llama-3.1-8b-instruct",
    "dataset": {"name": "test-dataset"},
    "hyperparameters": {
      "training_type": "sft",
      "finetuning_type": "lora",
      "epochs": 10,
      "batch_size": 16,
      "learning_rate": 0.0001,
      "lora": {
        "adapter_dim": 8,
      }
}'

Then view your results at wandb.ai under the nvidia-nemo-customizer project. W&B charts example

Note

The W&B integration is optional. When enabled, we’ll send training metrics to W&B using your API key. While we encrypt your API key and don’t log it internally, please review W&B’s terms of service before use.