Manage Metrics#

List, retrieve, and delete evaluation metrics using the NeMo Platform Python SDK.

import os
from nemo_platform import NeMoPlatform

client = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace="default",
)

List Metrics#

List all available evaluation metrics in a workspace, including both system-defined and custom user-defined metrics.

Basic Usage#

# List all metrics in the current workspace
metrics = client.evaluation.metrics.list()
for metric in metrics:
    print(f"{metric.name} ({metric.type}): {metric.description}")

Pagination#

Control the number of results returned per page and navigate through multiple pages of results.

# Get the first page with 50 metrics per page
metrics = client.evaluation.metrics.list(
    page=1,
    page_size=50
)

# Iterate through all pages
for metric in metrics:
    print(f"{metric.name}: {metric.description}")

# Get the second page
metrics_page_2 = client.evaluation.metrics.list(
    page=2,
    page_size=50
)

Sorting#

Sort metrics by different fields in ascending or descending order. Use - prefix for descending order.

# Sort by name (ascending)
metrics_by_name = client.evaluation.metrics.list(
    sort="name"
)

# Sort by creation date (most recent first)
metrics_recent = client.evaluation.metrics.list(
    sort="-created_at"
)

# Sort by update date (oldest first)
metrics_oldest_updated = client.evaluation.metrics.list(
    sort="updated_at"
)

Available sort fields:

name / -name: Sort by metric name
created_at / -created_at: Sort by creation timestamp
updated_at / -updated_at: Sort by last update timestamp

Discover Industry Metrics#

Industry metrics are predefined metrics available in the system workspace to use for your evaluation job. Discover all industry metrics or filter by label category.

Note

The system workspace is a reserved workspace for NeMo Platform that contains ready-to-use metrics representing industry metrics with published datasets.

metrics = client.evaluation.metrics.list(workspace="system")

print(f"{metrics.pagination.total_results} metrics")
for metric in metrics:
    print(f"{metric.name}: {metric.description}")

Filter by evaluation category level

metrics = client.evaluation.metrics.list(
    workspace="system",
    extra_query={"search": {"data.labels.eval_category": {"$eq": "agentic"}}},
)
for metric in metrics:
    print(f"{metric.name}: {metric.description}")

Retrieve a Specific Metric#

Get detailed information about a specific metric by its name.

# Retrieve a metric by name
metric = client.evaluation.metrics.retrieve(name="my-custom-metric")

print(f"Name: {metric.name}")
print(f"Type: {metric.type}")
print(f"Description: {metric.description}")

Search#

Search metrics using JSON search queries passed via extra_query. The search supports operators $eq, $like, $lt, $lte, $gt, $gte, $in, $nin and logical operators $and, $or, $not on fields such as name, description, type, project, created_at, and updated_at.

# Search by name
metrics = client.evaluation.metrics.list(
    extra_query={"search": {"name": {"$like": "bleu"}}}
)

# Search by type
metrics = client.evaluation.metrics.list(
    extra_query={"search": {"type": {"$eq": "llm-as-a-judge"}}}
)

# Combine multiple conditions
metrics = client.evaluation.metrics.list(
    extra_query={"search": {"$and": [{"name": {"$like": "bleu"}}, {"type": {"$eq": "similarity"}}]}}
)

# Search by date range
metrics = client.evaluation.metrics.list(
    extra_query={"search": {"created_at": {"$gte": "2025-01-01T00:00:00", "$lte": "2025-06-30T23:59:59"}}}
)

Delete a Metric#

Delete a custom evaluation metric. Industry metrics in the system workspace cannot be deleted.

Warning

Deleting a metric is permanent and cannot be undone. Ensure the metric is not being used by any active evaluations before deletion.

client.evaluation.metrics.delete(name="my-custom-metric")