Create Evaluation Target#

An evaluation target represents the subject of your evaluation—what you want to test or measure. Targets are reusable entities that can be evaluated multiple times with different configurations.

Target types include LLM models, RAG pipelines, retriever systems, and data sources (datasets or rows).

To create a target, send a POST request to the evaluation/targets API. The URL of the evaluator API depends on where you deploy evaluator and how you configure it. For more information, refer to NeMo Evaluator Deployment Guide.

Quick Links:

Resource	Description
Evaluation Concepts	Learn how targets fit into the evaluation workflow
Target Types Overview	Browse all available target types
Schema Reference	Complete field documentation

Prerequisites#

General Requirements#

Set your EVALUATOR_BASE_URL environment variable to your evaluator service endpoint:
```
export EVALUATOR_BASE_URL="https://your-evaluator-service-endpoint"
```
Determine your target namespace (defaults to "default" if not specified)

Target Type-Specific Requirements#

The requirements differ based on your target type. Choose the appropriate tab for detailed requirements:

Model Target

Use a model target to evaluate an LLM model endpoint.

Required:

A model resource with a configured api_endpoint that specifies:
- url – The model inference endpoint URL (e.g., http://nemo-nim-proxy:8000/v1/chat/completions)
- model_id – The model identifier (e.g., meta/llama-3.1-8b-instruct)
- format – API format ("nim" for NVIDIA NIM endpoints, "openai" for OpenAI-compatible endpoints)
The model endpoint must be network-accessible from the evaluator service
Valid authentication credentials if required by the model endpoint

Two Ways to Provide Model Information:

Reference an existing model by URN (e.g., "default/my-model-name")
Define the model inline with all required fields (see examples below)

Note

The evaluator validates model targets by making a test inference request. Ensure your model endpoint is reachable and responding before creating the target.

For more information, refer to LLM Model Targets.

RAG Target

Use a RAG target to evaluate an end-to-end Retrieval Augmented Generation pipeline combining document retrieval and LLM generation.

Required:

Generation model – A model for generating responses (URN or inline definition)
Retriever configuration – One of:
- Pipeline mode: Query embedding model, index embedding model, optional reranker model
- Cached mode: Pre-generated retriever outputs as a dataset
Dataset – Evaluation data source (for pipeline mode)

For more information, refer to RAG Pipeline Targets.

Retriever Target

Use a retriever target to evaluate document retrieval systems using embedding models and optional reranking.

Required:

Choose one of the following options:

Pipeline Mode:

Query embedding model (URN or inline definition)
Index embedding model (URN or inline definition)
Optional reranker model
Dataset with documents to index

Cached Outputs Mode:

Pre-generated retriever outputs as a dataset with files_url

For more information, refer to Retriever Pipeline Targets.

Rows Target

Use a rows target to specify evaluation data directly as an array of rows for quick testing or live evaluations.

Required:

An array of row objects containing evaluation data
Each row must include fields that match those referenced in your evaluation configuration (e.g., inputs, expected outputs)

Tip

Rows targets are useful for quick evaluations or testing without setting up a separate data source. They’re also the only target type supported for live evaluations via /v1/evaluation/live.

For more information, refer to Rows.

Dataset Target

Use a dataset target to reference an existing dataset resource for evaluation data or pre-generated model outputs.

Required:

A dataset resource with:
- files_url – Location of the dataset files (e.g., hf://namespace/dataset-name)
- Dataset must be accessible from the evaluator service

For more information, refer to Dataset.

To Create an Evaluation Target#

Choose one of the following options to create an evaluation target.

Python SDK

Example: Create Model Target with Inline Model Definition

import os
from nemo_microservices import NeMoMicroservices

# Initialize the client
client = NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL']
)

# Create an evaluation target with inline model definition
client.evaluation.targets.create(
    type="model",
    name="my-target-model-1",
    namespace="my-organization",
    model={
        "name": "model-MvPLX6aEa1zXJq7YMRCosm",
        "namespace": "default",
        "api_endpoint": {
            "url": "http://nemo-nim-proxy:8000/v1/chat/completions",
            "model_id": "meta/llama-3.1-8b-instruct",
            "format": "nim"
        }
    }
)

print("Target created successfully")

Example: Create Model Target with URN Reference

# Reference an existing model by URN
client.evaluation.targets.create(
    type="model",
    name="my-target-model-ref",
    namespace="my-organization",
    model="default/my-existing-model"  # URN reference to existing model
)

print("Target created successfully")

cURL

Example: Create Model Target with Inline Model Definition

curl -X "POST" "${EVALUATOR_BASE_URL}/v1/evaluation/targets" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "my-target-model-1",
    "namespace": "my-organization",
    "type": "model",
    "model": {
      "name": "model-MvPLX6aEa1zXJq7YMRCosm",
      "namespace": "default",
      "api_endpoint": {
        "url": "http://nemo-nim-proxy:8000/v1/chat/completions",
        "model_id": "meta/llama-3.1-8b-instruct",
        "format": "nim"
      }
    }
  }'

Example: Create Model Target with URN Reference

curl -X "POST" "${EVALUATOR_BASE_URL}/v1/evaluation/targets" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "my-target-model-ref",
    "namespace": "my-organization",
    "type": "model",
    "model": "default/my-existing-model"
  }'

Troubleshooting#

Error: “api_endpoint is not set”#

Cause: When creating a model target with an inline model definition, the api_endpoint field is required but was not provided.

Solution: Ensure your model object includes the api_endpoint field with url, model_id, and format:

model={
    "name": "my-model",
    "namespace": "default",
    "api_endpoint": {
        "url": "http://your-endpoint:8000/v1/chat/completions",
        "model_id": "your/model-id",
        "format": "nim"
    }
}

Error: “Model is not set”#

Cause: For model targets, the model field is required but was not provided.

Solution: Provide either a URN reference to an existing model or an inline model definition (see examples above).

Error: “Target type set to MODEL but other targets are present”#

Cause: When type is set to "model", only the model field should be populated. Other target fields (like rag, retriever, dataset, rows) must not be set.

Solution: Remove any other target fields and keep only the model field for model targets.

Advanced Options#

Skip Validation Checks#

By default, the evaluator validates targets during creation (e.g., tests model reachability). To skip validation checks during target creation, add the skip_validation_checks=true query parameter:

curl -X "POST" "${EVALUATOR_BASE_URL}/v1/evaluation/targets?skip_validation_checks=true" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{ ... }'

Warning

Skipping validation checks can result in evaluation jobs that fail to launch. Use this option only when you’re certain the target is valid or for testing purposes.