Bring Your Own Metric

NeMo Platform offers built-in metrics that can be configured to evaluate on your custom data. Remote metrics let you bring your own metric logic into the NeMo Platform evaluation workflow by serving that logic from a REST API.

A remote metric gives you control over the evaluation logic, request payload, and reported scores while the Evaluator plugin SDK handles dataset iteration, result aggregation, retries, and job execution.

Overview

Remote metrics support two types:

The Evaluator plugin SDK runs either type as a durable platform job through evaluator.submit(metric=metric, dataset=dataset). To iterate on a metric in-process without platform services, use the standalone nemo_evaluator_sdk.Evaluator instead.

Prerequisites

Before running remote metric evaluations:

1. Workspace: Have a workspace created.
2. Remote endpoint: Have your evaluation endpoint running and accessible.
3. API key (if required): If your endpoint requires authentication, create a secret to store the API key.
4. Initialize the SDK:

1
import os
2
3
from nemo_evaluator.sdk import Evaluator
4
from nemo_platform import NeMoPlatform
5
6
from nemo_evaluator_sdk import (
7
    NemoAgentToolkitRemoteMetric,
8
    RemoteMetric,
9
)
10
sdk = NeMoPlatform(
11
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
12
    workspace="default",
13
)
14
evaluator: Evaluator = sdk.evaluator  # this object is an Evaluator resource

Durable Remote Jobs

For production workloads, submit the same metric and dataset as a durable platform job. The returned job resource can wait for completion and download the final EvaluationResult.

1
from nemo_evaluator_sdk import RunConfig, JSONScoreParser, RemoteScore, RemoteMetric
2
3
metric = RemoteMetric(
4
    url="https://my-evaluation-server.test/evaluate",
5
    body={
6
        "reference": "{{item.reference}}",
7
        "response": "{{item.output}}",
8
    },
9
    scores=[
10
        RemoteScore(
11
            name="accuracy",
12
            parser=JSONScoreParser(json_path="$.result.accuracy"),
13
            minimum=0.0,
14
            maximum=1.0,
15
        )
16
    ],
17
    timeout_seconds=30.0,
18
    max_retries=3,
19
)
20
21
job = evaluator.submit(
22
    metric=metric,
23
    dataset=[
24
        {"reference": "Paris", "output": "Paris"},
25
        {"reference": "2", "output": "2"},
26
    ],
27
    config=RunConfig(parallelism=8),
28
)
29
print("Submitted job:", job.name)
30
31
job.wait_until_done()
32
result = job.get_result()
33
34
for score in result.aggregate_scores.scores:
35
    print(f"{score.name}: mean={score.mean}, count={score.count}")

Using API Key Secrets

If your remote endpoint requires authentication, store the API key as a platform secret and reference it from your metric:

For nemo_evaluator_sdk.Evaluator versus submitted-job behavior of api_key_secret, see Model API Authentication.

1
from nemo_evaluator_sdk import JSONScoreParser, RemoteScore, SecretRef, RemoteMetric
2
3
metric = RemoteMetric(
4
    url="https://my-authenticated-endpoint.test/evaluate",
5
    body={"input": "{{item.input}}"},
6
    scores=[RemoteScore(name="score", parser=JSONScoreParser(json_path="$.score"))],
7
    api_key_secret=SecretRef(root="my-remote-api-key"),
8
)
9
10
job = evaluator.submit(metric=metric, dataset=[{"input": "test"}])
11
job.wait_until_done()
12
result = job.get_result()

The API key is sent in the Authorization: Bearer <key> header. The job runtime resolves api_key_secret as a platform secret in the target workspace and receives it securely.

Endpoint Requirements

Your remote endpoint must:

1. Accept POST requests with Content-Type: application/json.
2. Return a JSON response containing the score values.

Example Endpoint (FastAPI)

1
from fastapi import FastAPI
2
from pydantic import BaseModel
3
4
app = FastAPI()
5
6
class MetricRequest(BaseModel):
7
    reference: str
8
    response: str
9
10
class MetricResponse(BaseModel):
11
    result: dict[str, float]
12
13
@app.post("/evaluate")
14
async def evaluate(request: MetricRequest) -> MetricResponse:
15
    accuracy = 1.0 if request.reference == request.response else 0.0
16
    return MetricResponse(result={"accuracy": accuracy})

NAT Endpoint Format

NAT endpoints receive:

1
{
2
  "evaluator_name": "similarity_eval",
3
  "item": {
4
    "id": "item_1",
5
    "input_obj": "What is the capital of France?",
6
    "expected_output_obj": "The capital of France is Paris.",
7
    "output_obj": "Paris is the capital.",
8
    "trajectory": [],
9
    "expected_trajectory": [],
10
    "full_dataset_entry": {}
11
  }
12
}

And must return:

1
{
2
  "success": true,
3
  "result": {
4
    "id": "item_1",
5
    "score": 0.85,
6
    "reasoning": {
7
      "method": "cosine_similarity"
8
    }
9
  },
10
  "error": null
11
}

The NAT metric extracts the score from $.result.score.

Configuration Options

Metric Parameters

Score Configuration (Generic Remote Only)

Each RemoteScore supports:

Parser configuration:

Example with all fields:

1
from nemo_evaluator_sdk import JSONScoreParser, RemoteScore
2
3
RemoteScore(
4
    name="accuracy",
5
    parser=JSONScoreParser(json_path="$.result.accuracy"),
6
    description="Measures response accuracy against reference",
7
    minimum=0.0,
8
    maximum=1.0,
9
)

Limitations

1. Network access: For job-based evaluation, remote metric endpoints must be reachable from the local platform runtime. Use a host or service URL that the platform can access.

2. Response format: Scores must be extractable using JSONPath from the response. Ensure your endpoint returns properly structured JSON.

3. Live evaluation limits: Live evaluations are limited to 10 rows. Use job-based evaluation for larger datasets.