Data Source Targets#

NeMo Evaluator provides two target types for specifying evaluation data directly:

Note

Data source targets (rows and dataset) are the only target types supported for live evaluations via /v1/evaluation/live. Other target types like model, retriever, or rag cannot be used with live evaluations.

Rows#

The rows target type allows you to directly specify evaluation data as an array of rows in the target definition. This is useful for quick evaluations or testing without needing to set up a separate data source.

Each row in the array should contain the input fields required by your evaluation configuration, such as inputs, expected outputs, or any other fields referenced in your metric definitions.

curl -X "POST" "${EVALUATOR_SERVICE_URL}/v1/evaluation/targets" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '
   {
      "type": "rows",
      "name": "my-target-rows-1",
      "namespace": "my-organization",
      "rows": [
         {
            "some_input": "Do you agree?",
            "some_output": "yes",
            "expected": "yes"
         }
         // Add more rows as needed
      ]
   }'

data = {
    "type": "rows",
    "name": "my-target-rows-1",
    "namespace": "my-organization",
    "rows": [
        {
            "some_input": "Do you agree?",
            "some_output": "yes", 
            "expected": "yes"
        }
        # Add more rows as needed
    ]
}

endpoint = f"{EVALUATOR_SERVICE_URL}/v1/evaluation/targets"

response = requests.post(endpoint, json=data).json()

Dataset#

The dataset target type allows you to reference evaluation data stored in NeMo Data Store or other supported dataset formats. This is useful for larger evaluations or when you want to maintain your evaluation data separately from your evaluation configuration.

curl -X "POST" "${EVALUATOR_SERVICE_URL}/v1/evaluation/targets" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '
   {
      "type": "dataset",
      "name": "my-target-dataset-1",
      "namespace": "my-organization",
      "dataset": {
         "files_url": "hf://datasets/<my-dataset-namespace>/<my-dataset-name>/<my-dataset-file-path>"
      }
   }'

data = {
    "type": "dataset",
    "name": "my-target-dataset-1",
    "namespace": "my-organization",
    "dataset": {
        "files_url": "hf://datasets/<my-dataset-namespace>/<my-dataset-name>/<my-dataset-file-path>"
    }
}

endpoint = f"{EVALUATOR_SERVICE_URL}/v1/evaluation/targets"

response = requests.post(endpoint, json=data).json()

Note

The dataset target also supports local file paths using the file:// prefix, which can be useful for testing and development.

On this page