Is this page helpful?

Filter, Search, and Sort Responses from the NVIDIA NeMo Evaluator API#

You can filter, search, and sort results when you query the NVIDIA NeMo Evaluator API. For example, you can list all evaluation jobs, filter for exact matches (like a specific namespace), search with substring matching (like jobs with names containing “training”), and sort by creation date. A sample response that includes information about filtering, searching, sorting, and pagination looks like the following.

{
    "object": "list",
    "data": [
        {
            // item details 
        },
        {
            // item details 
        },
        {
            // item details 
        }
    ],
    "pagination": {
        "page": 1,
        "page_size": 10,
        "current_page_size": 3,
        "total_pages": 1,
        "total_results": 3
    },
    "sort": "created_at",
    "filter": {
        "namespace": "my-organization",
        "created_at": {
            "gte": "2025-03-31T00:00:00"
        }
    },
    "search": {
        "name": ["training", "eval"]
    }
}

API Version Comparison#

The following table highlights key differences between v1 and v2 APIs for querying jobs:

Feature	v1 (Current)	v2 (Preview)
Status Values	`running`, `failed`, `ready`, `unknown`	`active`, `error`, `paused`, `pausing`, `resuming`
Date Operators	`gt`, `lt`	`gte`, `lte`
Sortable Fields	`created_at`, `updated_at`	`created_at`, `updated_at`
Filter Fields (exact match, single value)	`namespace`, `status`, `created_at` ✨ `target` (by namespace/name) ✨ `config` (by namespace/name)	`namespace`, `name`, `project`, `status`, `created_at`, `updated_at`
Search Fields (substring match, OR logic)	✨ `status`, ✨ `config`, ✨ `target`, `name`, `project`, and more	`name`, `project`

Note

To sort in descending order, prefix the field name with - (e.g., -created_at).

Understanding Filter vs Search Parameters#

When querying the API, you can use both filter and search parameters with different behaviors:

Filter Parameter#

Purpose: Exact matches on single values
Use Cases:
- Match exact namespace: filter[namespace]=my-organization
- Match exact status: filter[status]=completed
- Date ranges: filter[created_at][gte]=2025-03-18&filter[created_at][lte]=2025-03-20
Limitation: Does NOT support multiple values (e.g., cannot filter for multiple statuses at once)
Available Fields: Vary by API version (see comparison table above)

Search Parameter#

Purpose: Substring matching and OR logic with multiple values
Use Cases:
- Substring match: search[name]=train matches “training”, “train-model”, “retrain”
- Multiple values (OR logic): search[status]=running&search[status]=completed matches either status
- SDK syntax: search={'name': ['training', 'eval']}
Behavior: Case-insensitive substring matching when multiple values provided
Available Fields: Vary by API version (see comparison table above)

Tip

Use filter for precise matching (e.g., exact namespace), and search when you need flexibility (e.g., finding jobs with similar names or multiple statuses).

Filter Jobs (v1 - Current)#

You can filter Run and Manage Evaluation Jobs on the following fields. To see a sample response, refer to v2 (Preview).

Field	Type	Valid Values	Example
namespace	string	A namespace specified when a job was created.	`my-organization`
status	enum	One of: `created`, `pending`, `running`, `cancelled`, `cancelling`, `failed`, `completed`, `ready`, `unknown`.	`completed`
created_at	datetime	The date that a job was created, in ISO date format. You can use =, gt, lt.	`2025-03-18`
target	string	A target namespace and name (`namespace/name`).	`my-organization/my-model-target-1`
config	string	Config full name (`namespace/name`)	`my-organization/my-gsm8k-config-1`

Example: Filter Jobs by Single Status#

The following example gets all jobs with the exact status completed.

curl

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v1/evaluation/jobs?filter[status]=completed" \
  -H 'accept: application/json'

Python

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

response = client.evaluation.jobs.list(
    filter={'status': 'completed'}
)

Example: Filter Jobs by Date#

The following example gets all jobs created between March 18, 2025, and March 20, 2025.

curl

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v1/evaluation/jobs?filter[created_at][gt]=2025-03-18&filter[created_at][lt]=2025-03-20" \
  -H 'accept: application/json'

Python

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

response = client.evaluation.jobs.list(
    filter={'created_at': {'gt': '2025-03-18', 'lt': '2025-03-20'}}
)

Example: Filter Jobs by Target#

The following example gets all jobs that use a specific target.

curl

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v1/evaluation/jobs?filter[target]=my-organization/my-model-target-1" \
  -H 'accept: application/json'

Python

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

response = client.evaluation.jobs.list(
    filter={'target': 'my-organization/my-model-target-1'}
)

Example: Filter Jobs by Config#

The following example gets all jobs that use a specific config.

curl

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v1/evaluation/jobs?filter[config]=my-organization/my-gsm8k-config-1" \
  -H 'accept: application/json'

Python

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

response = client.evaluation.jobs.list(
    filter={'config': 'my-organization/my-gsm8k-config-1'}
)

Search Jobs (v1 - Current)#

You can search Run and Manage Evaluation Jobs using substring matching and OR logic for multiple values. Search is useful when you need to find jobs with similar names or match multiple status values.

Available search fields: status, config, target, name, project, and more.

Example: Search Jobs by Multiple Statuses#

The following example gets all jobs with status running OR completed (OR logic).

curl

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v1/evaluation/jobs?search[status]=running&search[status]=completed" \
  -H 'accept: application/json'

Python

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

response = client.evaluation.jobs.list(
    search={'status': ['running', 'completed']}
)

Example: Search Jobs by Name Substring#

The following example finds all jobs where the name contains “training” (case-insensitive substring match).

curl

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v1/evaluation/jobs?search[name]=training" \
  -H 'accept: application/json'

Python

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

response = client.evaluation.jobs.list(
    search={'name': 'training'}
)

Filter Jobs (v2 - Preview)#

Warning

v2 API Preview: The v2 API is available for testing and feedback but is not yet recommended for production use. Breaking changes may occur before the stable release.

You can filter Run and Manage Evaluation Jobs on the following fields. To see a sample response, refer to v2 (Preview).

Field	Type	Valid Values	Example
namespace	string	A namespace specified when a job was created.	`my-organization`
name	string	A name specified when a job was created.	`my-job-name`
project	string	A project specified when a job was created.	`my-project`
status	enum	One of: `created`, `pending`, `active`, `cancelled`, `cancelling`, `error`, `completed`, `paused`, `pausing`, `resuming`.	`completed`
created_at	datetime	The date that a job was created, in ISO date format. You can use gte, lte.	`2025-03-18`
updated_at	datetime	The date that a job was last updated, in ISO date format. You can use gte, lte.	`2025-03-18`

Example: Filter Jobs by Single Status#

The following example gets all jobs with the exact status completed.

curl

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v2/evaluation/jobs?filter[status]=completed" \
  -H 'accept: application/json'

Python

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

response = client.v2.evaluation.jobs.list(
    filter={'status': 'completed'}
)

Example: Filter Jobs by Date#

The following example gets all jobs created on or after March 18, 2025, and before or on March 20, 2025.

curl

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v2/evaluation/jobs?filter[created_at][gte]=2025-03-18&filter[created_at][lte]=2025-03-20" \
  -H 'accept: application/json'

Python

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

response = client.v2.evaluation.jobs.list(
    filter={'created_at': {'gte': '2025-03-18', 'lte': '2025-03-20'}}
)

Example: Filter Jobs by Name#

The following example gets all jobs with a specific name.

curl

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v2/evaluation/jobs?filter[name]=my-job-name" \
  -H 'accept: application/json'

Python

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

response = client.v2.evaluation.jobs.list(
    filter={'name': 'my-job-name'}
)

Example: Filter Jobs by Project#

The following example gets all jobs in a specific project.

curl

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v2/evaluation/jobs?filter[project]=my-project" \
  -H 'accept: application/json'

Python

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

response = client.v2.evaluation.jobs.list(
    filter={'project': 'my-project'}
)

Search Jobs (v2 - Preview)#

Warning

v2 API Preview: The v2 API is available for testing and feedback but is not yet recommended for production use. Breaking changes may occur before the stable release.

You can search Run and Manage Evaluation Jobs using substring matching and OR logic for multiple values.

Available search fields: name, project

Note

Unlike v1, v2 does not currently support searching by status. Use filter[status] for exact status matching.

Example: Search Jobs by Name Substring#

The following example finds all jobs where the name contains “training” (case-insensitive substring match).

curl

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v2/evaluation/jobs?search[name]=training" \
  -H 'accept: application/json'

Python

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

response = client.v2.evaluation.jobs.list(
    search={'name': 'training'}
)

Example: Search Jobs by Multiple Name Substrings#

The following example finds all jobs where the name contains “training” OR “eval” (OR logic).

curl

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v2/evaluation/jobs?search[name]=training&search[name]=eval" \
  -H 'accept: application/json'

Python

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

response = client.v2.evaluation.jobs.list(
    search={'name': ['training', 'eval']}
)

Filter Targets#

You can filter evaluation targets on the following fields. To see a sample response, refer to Create Evaluation Target.

Field	Type	Valid Values	Example
namespace	string	A namespace specified when a target was created.	`my-organization`
name	string	A name specified when a target was created.	`my-model-target-1`
type	string	`dataset`, `model`, `retriever`, `rag`, `cached_outputs` (deprecated)	`model`

Example: Filter Targets by Name#

The following example gets the target with the specified name.

curl

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v1/evaluation/targets?filter[namespace]=my-organization&filter[name]=my-model-target-1" \
  -H 'accept: application/json'

Python

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

response = client.evaluation.targets.list(
    filter={'namespace': 'my-organization', 'name': 'my-model-target-1'}
)

Filter Configs#

You can filter evaluation configurations on the following fields.

Field	Type	Valid Values	Example
namespace	string	A namespace specified when a config was created.	`my-organization`
name	string	A name specified when a config was created.	`my-gsm8k-config-1`
type	string	The type of configuration, such as custom, retriever, rag, gsm8k, and others.	`gsm8k`

Example: Filter Configs by Type#

The following example gets all configs of the specified type.

curl

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v1/evaluation/configs?filter[type]=gsm8k" \
  -H 'accept: application/json'

Python

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

response = client.evaluation.configs.list(
    filter={'type': 'gsm8k'}
)

Job Progress Information#

When you retrieve job status, the response includes progress tracking fields that help you monitor evaluation progress:

progress: Completion percentage (requires limit_samples in configuration)
samples_processed: Number of samples processed so far
task_status: Individual task status tracking

Example: Monitoring Job Progress#

The following example shows how to get job status with progress information.

v2 (Preview)#

In v2, status and progress information are consolidated into the main job details response.

curl

curl -X "GET" "${EVALUATOR_BASE_URL}/v2/evaluation/jobs/<job-id>" \
  -H 'accept: application/json'

Python

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

response = client.v2.evaluation.jobs.retrieve(job_id='<job-id>')
# Status and progress are included in the main response
print(f"Status: {response.status}")
print(f"Status details: {response.status_details}")

v1 (Current)#

curl

curl -X "GET" "${EVALUATOR_BASE_URL}/v1/evaluation/jobs/<job-id>/status" \
  -H 'accept: application/json'

Python

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

response = client.evaluation.jobs.status(job_id='<job-id>')

Response includes progress information:

{
    "message": "Job is running", 
    "progress": 42.5,
    "samples_processed": 85,
    "task_status": {
        "question-answering": "running"
    }
}

Sort API Responses#

You can sort job responses in ascending or descending order by created_at or updated_at.

Sortable Fields (both v1 and v2):

created_at - Sort by creation date (ascending)
-created_at - Sort by creation date (descending)
updated_at - Sort by last update date (ascending)
-updated_at - Sort by last update date (descending)

To sort results in descending order, prefix the field name with -.

Example: Sort Jobs#

The following example gets all jobs created on or after March 18, 2025, sorted by the date that the job was created in descending order.

v2 (Preview)#

curl

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v2/evaluation/jobs?filter[created_at][gte]=2025-03-18&sort=-created_at" \
  -H 'accept: application/json'

Python

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

response = client.v2.evaluation.jobs.list(
    filter={'created_at': {'gte': '2025-03-18'}},
    sort='-created_at'
)

v1 (Current)#

curl

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v1/evaluation/jobs?filter[created_at][gt]=2025-03-18&sort=-created_at" \
  -H 'accept: application/json'

Python

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

response = client.evaluation.jobs.list(
    filter={'created_at': {'gt': '2025-03-18'}},
    sort='-created_at'
)

Paginate API Responses#

By default, the API returns page 1, and uses a default page size of 10. You can request a particular page and adjust the page size as shown in the following example.

The following example gets all jobs created after a specific date, sets the page size to 5, and retrieves the information from page 3.

v2 (Preview)#

curl

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v2/evaluation/jobs?filter[created_at][gte]=2025-03-18&page_size=5&page=3" \
  -H 'accept: application/json'

Python

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

response = client.v2.evaluation.jobs.list(
    filter={'created_at': {'gte': '2025-03-18'}},
    page_size=5,
    page=3
)

v1 (Current)#

curl

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v1/evaluation/jobs?filter[created_at][gt]=2025-03-18&page_size=5&page=3" \
  -H 'accept: application/json'

Python

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

response = client.evaluation.jobs.list(
    filter={'created_at': {'gt': '2025-03-18'}},
    page_size=5,
    page=3
)

Advanced Examples#

Combining Multiple Filters#

You can combine multiple filters to narrow down your results:

curl (v2)

curl -X "GET" --globoff \
  "${EVALUATOR_BASE_URL}/v2/evaluation/jobs?filter[status]=completed&filter[project]=my-project&filter[created_at][gte]=2025-03-18" \
  -H 'accept: application/json'

Python (v2)

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

# Get completed jobs from a specific project created after a date
response = client.v2.evaluation.jobs.list(
    filter={
        'status': 'completed',
        'project': 'my-project',
        'created_at': {'gte': '2025-03-18'}
    }
)

Iterating Through All Pages#

To retrieve all jobs across multiple pages:

import os
import nemo_microservices

client = nemo_microservices.NeMoMicroservices(
    base_url=os.environ['EVALUATOR_BASE_URL'],
)

# Collect all jobs across pages
all_jobs = []
page = 1

while True:
    response = client.v2.evaluation.jobs.list(
        filter={'status': 'completed'},
        page=page,
        page_size=100
    )
    
    all_jobs.extend(response.data)
    
    if page >= response.pagination.total_pages:
        break
    
    page += 1

print(f"Retrieved {len(all_jobs)} total jobs")

Common Issues and Troubleshooting#

Date Filter Errors#

Problem: Getting a 422 error when filtering by date

Solution: Ensure you’re using the correct operators for your API version:

v1: Use gt (greater than) and lt (less than)
v2: Use gte (greater than or equal) and lte (less than or equal)

# ❌ Wrong for v2
?filter[created_at][gt]=2025-03-18

# ✅ Correct for v2
?filter[created_at][gte]=2025-03-18

Multiple Status Values Not Working#

Problem: Trying to filter by multiple statuses but getting errors

Solution: Use the search parameter instead of filter:

# ❌ This won't work
response = client.v2.evaluation.jobs.list(
    filter={'status': ['active', 'completed']}  # Error!
)

# ✅ Use search instead
response = client.v2.evaluation.jobs.list(
    search={'status': ['active', 'completed']}  # Works!
)

Job Not Found (404)#

Problem: Getting “Job not found” when retrieving a job

Solution: v1 and v2 jobs are separate. Ensure you’re using the correct API version:

# If job was created via v1 API, use v1 client
response = client.evaluation.jobs.retrieve(job_id='job-abc123')

# If job was created via v2 API, use v2 client
response = client.v2.evaluation.jobs.retrieve(job_id='job-abc123')

Understanding Job Status Values#

Problem: Filtering by running doesn’t return jobs in v2

Solution: v2 uses active instead of running. Check the API Version Comparison table for correct status values:

# v1
response = client.evaluation.jobs.list(search={'status': 'running'})

# v2
response = client.v2.evaluation.jobs.list(search={'status': 'active'})