LLM Model Targets#

An LLM model target points to a model, such as an LLM model, a chat endpoint, or a data file.

LLM Model Endpoint#

To create an evaluation target pointing to an LLM model running as NIM, specify a model that contains the api_endpoint of the model. For the list of NIM models, refer to Models.

To Create an LLM Model Endpoint Target#

Choose one of the following options to create an LLM model endpoint target.

import os
from nemo_microservices import NeMoMicroservices

# Initialize the client
client = NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL']
)

# Create an LLM model endpoint target
client.evaluation.targets.create(
    type="model",
    name="my-target-model-1",
    namespace="my-organization",
    model={
        "api_endpoint": {
            "url": "<my-nim-deployment-base-url>/completions",
            "model_id": "<my-model>"
        }
    }
)

print("Model target created successfully")

curl -X "POST" "${EVALUATOR_BASE_URL}/evaluation/targets" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
     "type": "model",
     "name": "my-target-model-1",
     "namespace": "my-organization",
     "model": {
        "api_endpoint": {
           "url": "<my-nim-deployment-base-url>/completions",
           "model_id": "<my-model>"
        }
     }
  }'

Chat Endpoints#

Unauthenticated#

To run an evaluation using an unauthenticated chat endpoint, specify a model.api_endpoint.url that contains a URL that ends with /chat/completions. This configuration is suitable for endpoints that don’t require authentication tokens or API keys.

To Create an Unauthenticated Chat Endpoint Target#

Choose one of the following options to create an unauthenticated chat endpoint target.

import os
from nemo_microservices import NeMoMicroservices

# Initialize the client
client = NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL']
)

# Create an unauthenticated chat endpoint target
client.evaluation.targets.create(
    type="model",
    name="my-target-model-2",
    namespace="my-organization",
    model={
        "api_endpoint": {
            "url": "<my-nim-deployment-base-url>/chat/completions",
            "model_id": "<my-model>"
        }
    }
)

print("Chat endpoint target created successfully")

curl -X "POST" "${EVALUATOR_BASE_URL}/evaluation/targets" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
     "type": "model",
     "name": "my-target-model-2",
     "namespace": "my-organization",
     "model": {
        "api_endpoint": {
           "url": "<my-nim-deployment-base-url>/chat/completions",
           "model_id": "<my-model>"
        }
     }
  }'

Authenticated (OpenAI)#

To run an evaluation on an authenticated endpoint that uses the OpenAI-compatible API format, specify openai for model.api_endpoint.format, and specify the API key for model.api_endpoint.api_key.

To Create an Authenticated OpenAI-Compatible Target#

Choose one of the following options to create an authenticated OpenAI-compatible target.

import os
from nemo_microservices import NeMoMicroservices

# Initialize the client
client = NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL']
)

# Create an authenticated OpenAI-compatible chat endpoint target
client.evaluation.targets.create(
    type="model",
    name="my-target-model-3",
    namespace="my-organization",
    model={
        "api_endpoint": {
            "url": "<external-openai-compatible-base-url>/chat/completions",
            "model_id": "<external-model>",
            "api_key": "<my-api-key>",
            "format": "openai"
        }
    }
)

print("Authenticated OpenAI target created successfully")

curl -X "POST" "${EVALUATOR_BASE_URL}/evaluation/targets" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
     "type": "model",
     "name": "my-target-model-3",
     "namespace": "my-organization",
     "model": {
        "api_endpoint": {
           "url": "<external-openai-compatible-base-url>/chat/completions",
           "model_id": "<external-model>",
           "api_key": "<my-api-key>",
           "format": "openai"        
        }
     }
  }'

Inference Parameters#

You can configure inference parameters that apply to all inference requests to the model target during evaluation.

{
  "type": "model",
  "model": {
    "api_endpoint": {
      "url": "<my-nim-deployment-base-url>/chat/completions",
      "model_id": "<my-model>"
    },
    "prompt": {
      "system_prompt": "",
      "inference_params": {
         "temperature": 0.6,
         "top_p": 0.95,
         "top_k": 1,
         "max_tokens": 1024,
         "stop": ["<|end_of_text|>", "<|eot|>"]
      }
    }
  }
}

Advanced Reasoning#

Advanced reasoning can be controlled for the target.

Nemotron#

Enable reasoning with Nemotron models with the system prompt 'detailed thinking on'. Nemotron models include reasoning steps in the output, denoted with <think></think> tags. Configure reasoning_params.end_token to omit reasoning tokens from evaluation.

{
  "type": "model",
  "model": {
    "api_endpoint": {
      "url": "<my-nim-deployment-base-url>/chat/completions",
      "model_id": "<my-model>"
    },
    "prompt": {
      "system_prompt": "'detailed thinking on'",
      "reasoning_params": {
        "end_token": "</think>"
      }
    }
  }
}

OpenAI#

{
  "type": "model",
  "model": {
    "api_endpoint": {
      "url": "<external-openai-compatible-base-url>/chat/completions",
      "model_id": "external-model",
      "api_key": "<my-api-key>",
      "format": "openai"
    },
    "prompt": {
      "reasoning_params": {
        "effort": "medium"
      }
    }
  }
}

Offline (Pre-generated)#

An offline (pre-generated) target points to a file that is stored in NeMo Data Store and that contains pre-generated answers. Offline targets are useful for similarity metrics evaluations. For more information, refer to Using Custom Data.

To Create an Offline (Pre-generated) Target#

Choose one of the following options to create an offline target.

import os
from nemo_microservices import NeMoMicroservices

# Initialize the client
client = NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL']
)

# Create an offline (pre-generated) target
client.evaluation.targets.create(
    type="cached_outputs",
    name="my-target-model-4",
    namespace="my-organization",
    cached_outputs={
        "files_url": "hf://datasets/<my-dataset-namespace>/<my-dataset-name>/<my-dataset-file-path>"
    }
)

print("Offline target created successfully")

curl -X "POST" "${EVALUATOR_BASE_URL}/evaluation/targets" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
     "type": "cached_outputs",
     "name": "my-target-model-4",
     "namespace": "my-organization",
     "cached_outputs": {
        "files_url": "hf://datasets/<my-dataset-namespace>/<my-dataset-name>/<my-dataset-file-path>"
     }
  }'