> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server.
# Checking Your Customization Job Metrics
After completing a customization job, you can monitor its performance through training and validation metrics. You can access these metrics in three ways:
1. Using the API
2. Through MLflow (optional)
3. Using Weights & Biases (optional)
The time to complete this tutorial is approximately 10 minutes.
## Prerequisites
#### New to using NeMo Platform?
All platform resources—models, datasets, and more—must belong to a **workspace**. Workspaces provide organizational and authorization boundaries for your work. Within a workspace, you can optionally use **projects** to group related resources.
**If you're new to the platform**, start with the **[Setup guide](/documentation/get-started)** to learn how to deploy and evaluate models, and optimize agents using the platform end-to-end.
**If you're already familiar** with workspaces and how to upload datasets to the platform, you can proceed directly with this tutorial.
For more information, see [Workspaces](/documentation/get-started/core-concepts/workspaces) and [Projects](/documentation/get-started/core-concepts/projects).
#### Platform Setup Requirements and Environment Variables
Before starting, make sure you have:
* NeMo Platform installed and deployed (see [Setup](/documentation/get-started))
* The PyPI `nemo-platform` wrapper package installed (`pip install "nemo-platform[all]"`). If you are working from a source checkout, run `make bootstrap` from the repository root instead.
* (Optional) Weights & Biases account and API key for enhanced visualization
**Set up environment variables:**
```bash
# Set the base URL for NeMo Platform
export NMP_BASE_URL="http://localhost:8080" # Or your deployed platform URL
# Optional: Weights & Biases for experiment tracking
export WANDB_API_KEY=""
```
**Initialize the SDK:**
```python
import os
from nemo_platform import NeMoPlatform
client = NeMoPlatform(
base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
workspace="default",
)
```
### Tutorial-Specific Prerequisites
* Completed customization job with a valid job name
* (Optional) A job created with `spec.integrations.mlflow` and access to its configured MLflow tracking server
## Available Metrics
Each customization job tracks two key metrics:
* **Training Loss**: Calculated during training, logged every 10 steps (default, configurable via hyperparameters)
* **Validation Loss**: Calculated during validation, logged at each validation interval
## Viewing Your Metrics
### Using the API
Get job status and training metrics through the platform Jobs service:
```python
import os
from nemo_platform import NeMoPlatform
client = NeMoPlatform(
base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
workspace="default",
)
# Get job status with metrics
job_name = "my-sft-job"
status = client.jobs.get_status(name=job_name, workspace="default")
print(f"Job: {status.name}")
print(f"Status: {status.status}")
# Check training step progress
for step in status.steps or []:
if step.name == "training":
for task in step.tasks or []:
details = task.status_details or {}
print(f"Training Phase: {details.get('phase')}")
print(f"Step: {details.get('step')}/{details.get('max_steps')}")
print(f"Epoch: {details.get('epoch')}/{details.get('num_epochs')}")
print(f"Training Loss: {details.get('train_loss')}")
print(f"Validation Loss: {details.get('val_loss')}")
print(f"Learning Rate: {details.get('lr')}")
print(f"Gradient Norm: {details.get('grad_norm')}")
```
The response includes training progress and metrics including loss, learning rate, and validation loss.
### Using MLflow
If your customization job was created with an `integrations.mlflow` configuration (see [MLflow Integration](/documentation/customizer-reference/manage-customization-jobs/customization-job-reference#mlflow-integration)):
1. Access the MLflow UI at the configured `tracking_uri`
2. Locate the configured `experiment_name` (defaults to the output model name)
3. Find the configured run `name` (defaults to the customization job ID)
4. View detailed metrics, including training and validation loss curves, under the "Metrics" tab
MLflow tracking is requested per job through `spec.integrations.mlflow`; it is not automatically enabled for every job in a cluster. The tracking server can be selected with `tracking_uri` in the job spec or the platform-side `MLFLOW_TRACKING_URI` environment variable. Contact your administrator if you need access to that server.
### Using Weights & Biases
If your customization job was created with W\&B integration enabled (see [Weights & Biases Integration](/documentation/customizer-reference/manage-customization-jobs/create-a-customization-job)):
1. Go to [wandb.ai](https://wandb.ai/home) and navigate to your project
2. Find the run corresponding to your customization job
3. View training and validation loss curves, learning rate schedules, and other metrics under the run's dashboard
```python
from nemo_automodel_plugin.schema import AutomodelJobInput
# Create an Automodel job with W&B integration
spec = AutomodelJobInput(
model="default/llama-3-2-1b",
dataset={"training": "default/my-dataset"},
training={"training_type": "sft", "finetuning_type": "lora"},
schedule={"epochs": 3},
batch={"global_batch_size": 16, "micro_batch_size": 1},
optimizer={"learning_rate": 1e-4},
integrations={
"wandb": {
"project": "my-finetuning-project",
"entity": "my-team",
"tags": ["fine-tuning", "llama"],
"api_key_secret": "my-wandb-key",
}
},
)
job = client.customization.automodel.jobs.create(
name="my-wandb-job",
workspace="default",
spec=spec,
)
print(f"Submitted job: {job.job.name}")
```
The `api_key_secret` field references a stored secret containing your `WANDB_API_KEY`.
Use the secret name (e.g., `"my-wandb-key"`) to resolve it from the request workspace.
To create the secret, see [Weights & Biases Keys](/documentation/get-started/core-concepts/manage-secrets).
Then view your results at [wandb.ai](https://wandb.ai/home) under your project.
The W\&B integration is optional and must be configured when [creating the customization job](/documentation/customizer-reference/manage-customization-jobs/create-a-customization-job). When enabled, training metrics are sent to W\&B using your API key. While we encrypt your API key and don't log it internally, please review W\&B's terms of service before use.