Data Source Targets#

NeMo Evaluator provides two target types for specifying evaluation data directly:

Note

Data source targets (rows and dataset) are the only target types supported for live evaluations via /evaluation/live. Other target types like model, retriever, or rag cannot be used with live evaluations.

Rows#

The rows target type allows you to directly specify evaluation data as an array of rows in the target definition. This is useful for quick evaluations or testing without needing to set up a separate data source.

Each row in the array should contain the input fields required by your evaluation configuration, such as inputs, expected outputs, or any other fields referenced in your metric definitions.

To Create a Rows Target#

Choose one of the following options to create a rows target.

import os
from nemo_microservices import NeMoMicroservices

# Initialize the client
client = NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL']
)

# Create a rows target
client.evaluation.targets.create(
    type="rows",
    name="my-target-rows-1",
    namespace="my-organization",
    rows=[
        {
            "some_input": "Do you agree?",
            "some_output": "yes",
            "expected": "yes"
        }
        # Add more rows as needed
    ]
)

print("Rows target created successfully")

curl -X "POST" "${EVALUATOR_BASE_URL}/evaluation/targets" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
     "type": "rows",
     "name": "my-target-rows-1",
     "namespace": "my-organization",
     "rows": [
        {
           "some_input": "Do you agree?",
           "some_output": "yes",
           "expected": "yes"
        }
     ]
  }'

Dataset#

The dataset target type allows you to reference evaluation data stored in NeMo Data Store or other supported dataset formats. This is useful for larger evaluations or when you want to maintain your evaluation data separately from your evaluation configuration.

To Create a Dataset Target#

Choose one of the following options to create a dataset target.

import os
from nemo_microservices import NeMoMicroservices

# Initialize the client
client = NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL']
)

# Create a dataset target
client.evaluation.targets.create(
    type="dataset",
    name="my-target-dataset-1",
    namespace="my-organization",
    dataset={
        "files_url": "hf://datasets/<my-dataset-namespace>/<my-dataset-name>/<my-dataset-file-path>"
    }
)

print("Dataset target created successfully")

curl -X "POST" "${EVALUATOR_BASE_URL}/evaluation/targets" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
     "type": "dataset",
     "name": "my-target-dataset-1",  
     "namespace": "my-organization",
     "dataset": {
        "files_url": "hf://datasets/<my-dataset-namespace>/<my-dataset-name>/<my-dataset-file-path>"
     }
  }'

Note

The dataset target also supports local file paths using the file:// prefix, which can be useful for testing and development.