Filter and Sort Responses from the NVIDIA NeMo Evaluator API#

You can filter and sort results when you query the NVIDIA NeMo Evaluator API. For example, you can list all evaluation jobs, but use a filter to return only jobs that were created today. A sample response that includes information about filtering, sorting, and pagination looks like the following.

{
    "object": "list",
    "data": [
        {
            // item details 
        },
        {
            // item details 
        },
        {
            // item details 
        }
    ],
    "pagination": {
        "page": 1,
        "page_size": 10,
        "current_page_size": 3,
        "total_pages": 1,
        "total_results": 3
    },
    "sort": "created_at",
    "filter": {
        "created_at": {
            "gt": "2025-03-31T00:00:00"
        }
    }
}

Filter Jobs#

You can filter Run and Manage Evaluation Jobs on the following fields. To see a sample response, refer to v2 (Preview).

Field

Type

Valid Values

Example

namespace

string

A namespace specified when a job was created.

my-organization

status

enum

One of: created, pending, running, completed, failed.

completed

created_at

datetime

The date that a job was created, in ISO date format. You can use =, gt, lt.

2025-03-18

target

string

A target namespace and name (namespace/name).

my-organization/my-model-target-1

config

string

Config full name (namespace/name)

my-organization/my-gsm8k-config-1

Example: Filter Jobs by Status#

The following example gets all jobs with the status running.

v2 (Preview)#

Warning

v2 API Preview: The v2 API is available for testing and feedback but is not yet recommended for production use. Breaking changes may occur before the stable release.

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v2/evaluation/jobs?filter[status]=running" \
  -H 'accept: application/json'

import os
import requests

endpoint = f"{os.environ['EVALUATOR_BASE_URL']}/v2/evaluation/jobs?filter[status]=running"
response = requests.get(endpoint).json()
response

v1 (Current)#

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v1/evaluation/jobs?filter[status]=running" \
  -H 'accept: application/json'

import os
import requests

endpoint = f"{os.environ['EVALUATOR_BASE_URL']}/v1/evaluation/jobs?filter[status]=running"
response = requests.get(endpoint).json()
response

Example: Filter Jobs by Date#

The following example gets all jobs created between March 18, 2025, and March 20, 2025.

v2 (Preview)#

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v2/evaluation/jobs?filter[created_at][gt]=2025-03-18&filter[created_at][lt]=2025-03-20" \
  -H 'accept: application/json'

endpoint = f"{os.environ['EVALUATOR_BASE_URL']}/v2/evaluation/jobs?filter[created_at][gt]=2025-03-18&filter[created_at][lt]=2025-03-20"
response = requests.get(endpoint).json()
response

v1 (Current)#

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v1/evaluation/jobs?filter[created_at][gt]=2025-03-18&filter[created_at][lt]=2025-03-20" \
  -H 'accept: application/json'

endpoint = f"{os.environ['EVALUATOR_BASE_URL']}/v1/evaluation/jobs?filter[created_at][gt]=2025-03-18&filter[created_at][lt]=2025-03-20"
response = requests.get(endpoint).json()
response

Example: Filter Jobs by Target#

The following example gets all jobs that use a specific target.

v2 (Preview)#

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v2/evaluation/jobs?filter[target]=my-organization/my-model-target-1" \
  -H 'accept: application/json'

endpoint = f"{os.environ['EVALUATOR_BASE_URL']}/v2/evaluation/jobs?filter[target]=my-organization/my-model-target-1"
response = requests.get(endpoint).json()
response

v1 (Current)#

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v1/evaluation/jobs?filter[target]=my-organization/my-model-target-1" \
  -H 'accept: application/json'

endpoint = f"{os.environ['EVALUATOR_BASE_URL']}/v1/evaluation/jobs?filter[target]=my-organization/my-model-target-1"
response = requests.get(endpoint).json()
response

Example: Filter Jobs by Config#

The following example gets all jobs that use a specific config.

v2 (Preview)#

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v2/evaluation/jobs?filter[config]=my-organization/my-gsm8k-config-1" \
  -H 'accept: application/json'

endpoint = f"{os.environ['EVALUATOR_BASE_URL']}/v2/evaluation/jobs?filter[config]=my-organization/my-gsm8k-config-1"
response = requests.get(endpoint).json()
response

v1 (Current)#

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v1/evaluation/jobs?filter[config]=my-organization/my-gsm8k-config-1" \
  -H 'accept: application/json'

endpoint = f"{os.environ['EVALUATOR_BASE_URL']}/v1/evaluation/jobs?filter[config]=my-organization/my-gsm8k-config-1"
response = requests.get(endpoint).json()
response

Filter Targets#

You can filter evaluation targets on the following fields. To see a sample response, refer to Create Evaluation Target.

Field

Type

Valid Values

Example

namespace

string

A namespace specified when a target was created.

my-organization

name

string

A name specified when a target was created.

my-model-target-1

type

string

cached_outputs, model, retriever, rag

model

Example: Filter Targets by Name#

The following example gets the target with the specified name.

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v1/evaluation/targets?filter[namespace]=my-organization&filter[name]=my-model-target-1" \
  -H 'accept: application/json'

endpoint = f"{os.environ['EVALUATOR_BASE_URL']}/v1/evaluation/targets?filter[namespace]=my-organization&filter[name]=my-model-target-1"
response = requests.get(endpoint).json()
response

Filter Configs#

You can filter evaluation configurations on the following fields.

Field

Type

Valid Values

Example

namespace

string

A namespace specified when a config was created.

my-organization

name

string

A name specified when a config was created.

my-gsm8k-config-1

type

string

The type of configuration, such as custom, retriever, rag, gsm8k, and others.

gsm8k

Example: Filter Configs by Type#

The following example gets all configs of the specified type.

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v1/evaluation/configs?filter[type]=gsm8k" \
  -H 'accept: application/json'

endpoint = f"{os.environ['EVALUATOR_BASE_URL']}/v1/evaluation/configs?filter[type]=gsm8k"
response = requests.get(endpoint).json()
response

Job Progress Information#

When you retrieve job status, the response includes progress tracking fields that help you monitor evaluation progress:

  • progress: Completion percentage (requires limit_samples in configuration)

  • samples_processed: Number of samples processed so far

  • task_status: Individual task status tracking

Example: Monitoring Job Progress#

The following example shows how to get job status with progress information.

v2 (Preview)#

In v2, status and progress information are consolidated into the main job details response.

curl -X "GET" "${EVALUATOR_BASE_URL}/v2/evaluation/jobs/<job-id>" \
  -H 'accept: application/json'

endpoint = f"{os.environ['EVALUATOR_BASE_URL']}/v2/evaluation/jobs/<job-id>"
response = requests.get(endpoint).json()
# Status and progress are included in the main response
print(f"Status: {response['status']}")
print(f"Status details: {response['status_details']}")
response

v1 (Current)#

curl -X "GET" "${EVALUATOR_BASE_URL}/v1/evaluation/jobs/<job-id>/status" \
  -H 'accept: application/json'

endpoint = f"{os.environ['EVALUATOR_BASE_URL']}/v1/evaluation/jobs/<job-id>/status"
response = requests.get(endpoint).json()
response

Response includes progress information:

{
    "message": "Job is running", 
    "progress": 42.5,
    "samples_processed": 85,
    "task_status": {
        "question-answering": "running"
    }
}

Sort API Responses#

You can sort responses in ascending or descending order. To sort results in descending order, put - in front of the sort field.

The following example gets all jobs created after March 18, 2025, sorted by the date that the job was created in descending order.

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v1/evaluation/jobs?filter[created_at][gt]=2025-03-18&sort=-created_at" \
  -H 'accept: application/json'

endpoint = f"{os.environ['EVALUATOR_BASE_URL']}/v1/evaluation/jobs?filter[created_at][gt]=2025-03-18&sort=-created_at"
response = requests.get(endpoint).json()
response

Paginate API Responses#

By default, the API returns page 1, and uses a default page size of 10. You can request a particular page and adjust the page size as shown in the following example.

The following example gets all jobs created after March 18, 2025, sets the page size to 5, and retrieves the information from page 3.

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v1/evaluation/jobs?filter[created_at][gt]=2025-03-18&page_size=5&page=3" \
  -H 'accept: application/json'

endpoint = f"{os.environ['EVALUATOR_BASE_URL']}/v1/evaluation/jobs?filter[created_at][gt]=2025-03-18&page_size=5&page=3"
response = requests.get(endpoint).json()
response