Create Evaluation Job#

To create an evaluation job, send a POST request to the evaluation/jobs API. The URL of the evaluator API depends on where you deploy evaluator and how you configure it. For more information, refer to Deploy the NeMo Evaluator Microservice.

Prerequisites#

  • Set your EVALUATOR_BASE_URL environment variable to your evaluator service endpoint:

    export EVALUATOR_BASE_URL="https://your-evaluator-service-endpoint"
    
  • Ensure you have created both an evaluation target and an evaluation configuration

To Create an Evaluation Job#

Choose one of the following options to create an evaluation job.

import os
from nemo_microservices import NeMoMicroservices

# Initialize the client
client = NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL']
)

# Create an evaluation job
job = client.evaluation.jobs.create(
    namespace="my-organization",
    target={
        "type": "model",
        "name": "my-target-name",
        "namespace": "my-target-namespace"
    },
    config={
        "type": "custom", 
        "name": "my-config-name",
        "namespace": "my-config-namespace"
    }
)

# Get the job ID and status
job_id = job.id
print(f"Job ID: {job_id}")
print(f"Job status: {job.status}")
curl -X "POST" "${EVALUATOR_BASE_URL}/evaluation/jobs" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "namespace": "my-organization",
    "target": "<my-target-namespace/my-target-name>",
    "config": "<my-config-namespace/my-config-name>"
  }'
Example Response
{
    "created_at": "2025-03-19T22:50:15.684382",
    "updated_at": "2025-03-19T22:50:15.684385",
    "id": "eval-UVW123XYZ456",
    "namespace": "my-organization",
    "description": null,
    "target": {
        //target details
    },
    "config": {
        // config details
    },
    "result": null,
    "output_files_url": null,
    "status_details": {
        "message": null,
        "task_status": {},
        "progress": null
    },
    "status": "created",
    "project": null,
    "custom_fields": {},
    "ownership": null
}

For the full response reference, refer to Evaluator API.