LLM Model Targets

An LLM model target points to a model, such as an LLM model, a chat endpoint, or a data file.

LLM Model Endpoint

To create an evaluation target pointing to an LLM model running as NIM, specify a model that contains the api_endpoint of the model. For the list of NIM models, refer to Models.

curl

curl -X "POST" "${EVALUATOR_SERVICE_URL}/v1/evaluation/targets" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '
   {
      "type": "model",
      "name": "my-target-model-1",
      "namespace": "my-organization",
      "model": {
         "api_endpoint": {
            "url": "<my-nim-deployment-base-url>/completions",
            "model_id": "<my-model>"
         }
      }
   }'

Python

data = {
    "type": "model",
    "name": "my-target-model-1",
    "namespace": "my-organization",
    "model": {
        "api_endpoint": {
        "url": "<my-nim-deployment-base-url>/completions",
        "model_id": "<my-model>"
        }
    }
}

endpoint = f"{EVALUATOR_SERVICE_URL}/v1/evaluation/targets"

response = requests.post(endpoint, json=data).json()

Chat Endpoints

Unauthenticated

To run an evaluation using an unauthenticated chat endpoint, specify a model.api_endpoint.url that contains a URL that ends with /chat/completions. This configuration is suitable for endpoints that don’t require authentication tokens or API keys.

curl

curl -X "POST" "${EVALUATOR_SERVICE_URL}/v1/evaluation/targets" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '
   {
      "type": "model",
      "name": "my-target-model-2",
      "namespace": "my-organization",
      "model": {
         "api_endpoint": {
            "url": "<my-nim-deployment-base-url>/chat/completions",
            "model_id": "<my-model>"
         }
      }
   }'

Python

data = {
    "type": "model",
    "name": "my-target-model-2",
    "namespace": "my-organization",
    "model": {
        "api_endpoint": {
        "url": "<my-nim-deployment-base-url>/chat/completions",
        "model_id": "<my-model>"
        }
    }
}

endpoint = f"{EVALUATOR_SERVICE_URL}/v1/evaluation/targets"

response = requests.post(endpoint, json=data).json()

Authenticated (OpenAI)

To run an evaluation on an authenticated endpoint that uses the OpenAI-compatible API format, specify openai for model.api_endpoint.format, and specify the API key for model.api_endpoint.api_key.

curl

curl -X "POST" "${EVALUATOR_SERVICE_URL}/v1/evaluation/targets" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '
   {
      "type": "model",
      "name": "my-target-model-3",
      "namespace": "my-organization",
      "model": {
         "api_endpoint": {
            "url": "<external-openai-compatible-base-url>/chat/completions",
            "model_id": "<external-model>",
            "api_key": "<my-api-key>",
            "format": "openai"        
         }
      }
   }'

Python

data = {
    "type": "model",
    "name": "my-target-model-3",
    "namespace": "my-organization",
    "model": {
        "api_endpoint": {
        "url": "<external-openai-compatible-base-url>/chat/completions",
        "model_id": "<external-model>",
        "api_key": "<my-api-key>",
        "format": "openai"        
        }
    }
}

endpoint = f"{EVALUATOR_SERVICE_URL}/v1/evaluation/targets"

response = requests.post(endpoint, json=data).json()

Offline (Pre-generated)

An offline (pre-generated) target points to a file that is stored in NeMo Data Store and that contains pre-generated answers. Offline targets are useful for similarity metrics evaluations. For more information, refer to Using Custom Data.

curl

curl -X "POST" "${EVALUATOR_SERVICE_URL}/v1/evaluation/targets" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '
   {
      "type": "cached_outputs",
      "name": "my-target-model-4",
      "namespace": "my-organization",
      "cached_outputs": {
         "files_url": "hf://datasets/<my-dataset-namespace>/<my-dataset-name>/<my-dataset-file-path>"
      }
   }'

Python

data = {
    "type": "cached_outputs",
    "name": "my-target-model-4",
    "namespace": "my-organization",
    "cached_outputs": {
        "files_url": "hf://datasets/<my-dataset-namespace>/<my-dataset-name>/<my-dataset-file-path>"
    }
}

endpoint = f"{EVALUATOR_SERVICE_URL}/v1/evaluation/targets"

response = requests.post(endpoint, json=data).json()