Create Evaluation Job#
To create an evaluation job, send a POST
request to the evaluation/jobs
API. The URL of the evaluator API depends on where you deploy evaluator and how you configure it. For more information, refer to Deploy the NeMo Evaluator Microservice.
Prerequisites#
Set your
EVALUATOR_BASE_URL
environment variable to your evaluator service endpoint:export EVALUATOR_BASE_URL="https://your-evaluator-service-endpoint"
Ensure you have created both an evaluation target and an evaluation configuration
To Create an Evaluation Job#
Choose one of the following options to create an evaluation job.
import os
from nemo_microservices import NeMoMicroservices
# Initialize the client
client = NeMoMicroservices(
base_url=os.environ['EVALUATOR_BASE_URL']
)
# Create an evaluation job
job = client.evaluation.jobs.create(
namespace="my-organization",
target={
"type": "model",
"name": "my-target-name",
"namespace": "my-target-namespace"
},
config={
"type": "custom",
"name": "my-config-name",
"namespace": "my-config-namespace"
}
)
# Get the job ID and status
job_id = job.id
print(f"Job ID: {job_id}")
print(f"Job status: {job.status}")
curl -X "POST" "${EVALUATOR_BASE_URL}/evaluation/jobs" \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"namespace": "my-organization",
"target": "<my-target-namespace/my-target-name>",
"config": "<my-config-namespace/my-config-name>"
}'
Example Response
{
"created_at": "2025-03-19T22:50:15.684382",
"updated_at": "2025-03-19T22:50:15.684385",
"id": "eval-UVW123XYZ456",
"namespace": "my-organization",
"description": null,
"target": {
//target details
},
"config": {
// config details
},
"result": null,
"output_files_url": null,
"status_details": {
"message": null,
"task_status": {},
"progress": null
},
"status": "created",
"project": null,
"custom_fields": {},
"ownership": null
}
For the full response reference, refer to Evaluator API.