Create Evaluation Job#

To create an evaluation job, send a POST request to the evaluation/jobs API. The URL of the evaluator API depends on where you deploy evaluator and how you configure it. For more information, refer to Deploy the NeMo Evaluator Microservice.

Prerequisites#

The examples in this documentation specify {EVALUATOR_SERVICE_URL} in the code. Do the following to store the evaluator hostname to use it in your code.

Important

Replace <your evaluator service endpoint> with your address, such as https://evaluator.internal.your-company.com, before you run this code.

export EVALUATOR_SERVICE_URL="<your evaluator service endpoint>"

import requests

EVALUATOR_SERVICE_URL = "<your evaluator service endpoint>"

Options#

API#

Use the following code to create an evaluation job.

curl -X "POST" "${EVALUATOR_SERVICE_URL}/v1/evaluation/jobs" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '
    {
      "namespace": "my-organization",
      "target": "<my-target-namespace/my-target-name>",
      "config": "<my-config-namespace/my-config-name>"
    }'

data = {
   "namespace": "my-organization",
   "target": "<my-target-namespace/my-target-name>",
   "config": "<my-config-namespace/my-config-name>"
}

endpoint = f"{EVALUATOR_SERVICE_URL}/v1/evaluation/jobs"

# Make the API call
response = requests.post(endpoint, json=data).json()

# Get the job_id so we can refer to it later
job_id = response['id']
print(f"Job ID: {job_id}")

# Get the status.  You should see `CREATED` or `PENDING`, or `RUNNING`.
job_status = response['status']
print(f"Job status: {job_status}")
Example Response
{
    "created_at": "2025-03-19T22:50:15.684382",
    "updated_at": "2025-03-19T22:50:15.684385",
    "id": "eval-UVW123XYZ456",
    "namespace": "my-organization",
    "description": null,
    "target": {
        //target details
    },
    "config": {
        // config details
    },
    "result": null,
    "output_files_url": null,
    "status_details": {
        "message": null,
        "task_status": {},
        "progress": null
    },
    "status": "created",
    "project": null,
    "custom_fields": {},
    "ownership": null
}

For the full response reference, refer to Evaluator API.

On this page