`nemo_microservices.resources.evaluation.jobs`#

Module Contents#

Classes#

`AsyncJobsResource`
`AsyncJobsResourceWithRawResponse`
`AsyncJobsResourceWithStreamingResponse`
`JobsResource`
`JobsResourceWithRawResponse`
`JobsResourceWithStreamingResponse`

API#

class nemo_microservices.resources.evaluation.jobs.AsyncJobsResource( client: nemo_microservices._client.AsyncNeMoMicroservices, )#

Bases: nemo_microservices._resource.AsyncAPIResource

Initialization

async cancel( job_id: str, *, extra_headers: nemo_microservices._types.Headers | None = None, extra_query: nemo_microservices._types.Query | None = None, extra_body: nemo_microservices._types.Body | None = None, timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given, ) → object#

Cancel an evaluation job.

Args: extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

async create( *, config: nemo_microservices.types.evaluation.job_create_params.Config, target: nemo_microservices.types.evaluation.job_create_params.Target, custom_fields: Dict[str, object] | nemo_microservices._types.Omit = omit, description: str | nemo_microservices._types.Omit = omit, namespace: str | nemo_microservices._types.Omit = omit, output_files_url: str | nemo_microservices._types.Omit = omit, ownership: nemo_microservices.types.shared_params.ownership.Ownership | nemo_microservices._types.Omit = omit, project: str | nemo_microservices._types.Omit = omit, result: str | nemo_microservices._types.Omit = omit, status: nemo_microservices.types.shared.job_status.JobStatus | nemo_microservices._types.Omit = omit, status_details: nemo_microservices.types.evaluation_status_details_param.EvaluationStatusDetailsParam | nemo_microservices._types.Omit = omit, extra_headers: nemo_microservices._types.Headers | None = None, extra_query: nemo_microservices._types.Query | None = None, extra_body: nemo_microservices._types.Body | None = None, timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given, ) → nemo_microservices.types.evaluation.evaluation_job.EvaluationJob#

Start a new evaluation job by providing the necessary model and evaluation specifications.

A unique id will be created and provided in the response along with additional details.

Args: config: The evaluation configuration.

target: The evaluation target.

custom_fields: A set of custom fields that the user can define and use for various purposes.

description: The description of the entity.

namespace: The namespace of the entity. This can be missing for namespace entities or in deployments that don’t use namespaces.

output_files_url: The place where the output files, if any, should be written.

ownership: Information about ownership of an entity.

  If the entity is a namespace, the `access_policies` will typically apply to all
  entities inside the namespace.

project: The URN of the project associated with this entity.

result: The evaluation result URN.

status: Normalized statuses for all jobs.

  - **CREATED**: The job is created, but not yet scheduled.
  - **PENDING**: The job is waiting for resource allocation.
  - **RUNNING**: The job is currently running.
  - **CANCELLING**: The job is being cancelled at the user's request.
  - **CANCELLED**: The job has been cancelled by the user.
  - **CANCELLING**: The job is being cancelled at the user's request.
  - **FAILED**: The job failed to execute and terminated.
  - **COMPLETED**: The job has completed successfully.
  - **READY**: The job is ready to be used.
  - **UNKNOWN**: The job status is unknown.

status_details: Details about the status of the evaluation.

extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

async delete( job_id: str, *, extra_headers: nemo_microservices._types.Headers | None = None, extra_query: nemo_microservices._types.Query | None = None, extra_body: nemo_microservices._types.Body | None = None, timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given, ) → None#

Delete Evaluation by Evaluation ID.

Args: extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

async download_logs( job_id: str, *, extra_headers: nemo_microservices._types.Headers | None = None, extra_query: nemo_microservices._types.Query | None = None, extra_body: nemo_microservices._types.Body | None = None, timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given, ) → object#

Return the logs generated by the evaluation job.

The logs are returned as a ZIP file.

Args: extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

async download_results( job_id: str, *, extra_headers: nemo_microservices._types.Headers | None = None, extra_query: nemo_microservices._types.Query | None = None, extra_body: nemo_microservices._types.Body | None = None, timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given, ) → nemo_microservices._response.AsyncBinaryAPIResponse#

Download the result directory containing configuration files, logs, and evaluation results for a specific evaluation run. The result directory is packaged and provided as a downloadable archive.

Args: extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

List available evaluation jobs.

Args: filter: Filter jobs on various criteria.

page: Page number.

page_size: Page size.

search: Search evaluation jobs using substring matching. You can combine multiple search fields and filters.

  For example:

  - `?search[status]=running`: searches all jobs with 'running' in the status.
  - `?search[config]=llama-benchmark`: searches all jobs with 'llama-benchmark' in
    the config field.
  - `?search[target]=model-accuracy`: searches all jobs with 'model-accuracy' in
    the target field.
  - `?search[status]=running&search[config]=llama-benchmark`: searches all jobs
    with 'running' in the status AND 'llama-benchmark' in the config field.
  - `?search[updated_at][start]=2024-01-01T00:00:00` finds all jobs updated on or
    after the start date
  - `?search[created_at][start]=2022-01-01&search[updated_at][end]=2024-01-01`
    finds all jobs created from start date up to and including end date

sort: The field to sort by. To sort in decreasing order, use - in front of the field name.

extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

async results( job_id: str, *, extra_headers: nemo_microservices._types.Headers | None = None, extra_query: nemo_microservices._types.Query | None = None, extra_body: nemo_microservices._types.Body | None = None, timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given, ) → nemo_microservices.types.evaluation_result.EvaluationResult#

Return the results of the evaluation.

Args: extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

async retrieve( job_id: str, *, extra_headers: nemo_microservices._types.Headers | None = None, extra_query: nemo_microservices._types.Query | None = None, extra_body: nemo_microservices._types.Body | None = None, timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given, ) → nemo_microservices.types.evaluation.evaluation_job.EvaluationJob#

Access detailed information about a specific evaluation.

Once an evaluation is finished, retrieve comprehensive results.

Args: extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

async status( job_id: str, *, extra_headers: nemo_microservices._types.Headers | None = None, extra_query: nemo_microservices._types.Query | None = None, extra_body: nemo_microservices._types.Body | None = None, timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given, ) → nemo_microservices.types.evaluation_status_details.EvaluationStatusDetails#

Access detailed information about a specific evaluation.

Once an evaluation is finished, retrieve comprehensive results.

Args: extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

property with_raw_response: nemo_microservices.resources.evaluation.jobs.AsyncJobsResourceWithRawResponse#

This property can be used as a prefix for any HTTP method call to return the raw response object instead of the parsed content.

For more information, see https://docs.nvidia.com/nemo/microservices/latest/pysdk/index.html#accessing-raw-response-data-e-g-headers

property with_streaming_response: nemo_microservices.resources.evaluation.jobs.AsyncJobsResourceWithStreamingResponse#

An alternative to .with_raw_response that doesn’t eagerly read the response body.

For more information, see https://docs.nvidia.com/nemo/microservices/latest/pysdk/index.html#with_streaming_response

class nemo_microservices.resources.evaluation.jobs.AsyncJobsResourceWithRawResponse( jobs: nemo_microservices.resources.evaluation.jobs.AsyncJobsResource, )#: Initialization

class nemo_microservices.resources.evaluation.jobs.AsyncJobsResourceWithStreamingResponse( jobs: nemo_microservices.resources.evaluation.jobs.AsyncJobsResource, )#: Initialization

class nemo_microservices.resources.evaluation.jobs.JobsResource(client: nemo_microservices._client.NeMoMicroservices)#

Bases: nemo_microservices._resource.SyncAPIResource

Initialization

cancel( job_id: str, *, extra_headers: nemo_microservices._types.Headers | None = None, extra_query: nemo_microservices._types.Query | None = None, extra_body: nemo_microservices._types.Body | None = None, timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given, ) → object#

Cancel an evaluation job.

Args: extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

create( *, config: nemo_microservices.types.evaluation.job_create_params.Config, target: nemo_microservices.types.evaluation.job_create_params.Target, custom_fields: Dict[str, object] | nemo_microservices._types.Omit = omit, description: str | nemo_microservices._types.Omit = omit, namespace: str | nemo_microservices._types.Omit = omit, output_files_url: str | nemo_microservices._types.Omit = omit, ownership: nemo_microservices.types.shared_params.ownership.Ownership | nemo_microservices._types.Omit = omit, project: str | nemo_microservices._types.Omit = omit, result: str | nemo_microservices._types.Omit = omit, status: nemo_microservices.types.shared.job_status.JobStatus | nemo_microservices._types.Omit = omit, status_details: nemo_microservices.types.evaluation_status_details_param.EvaluationStatusDetailsParam | nemo_microservices._types.Omit = omit, extra_headers: nemo_microservices._types.Headers | None = None, extra_query: nemo_microservices._types.Query | None = None, extra_body: nemo_microservices._types.Body | None = None, timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given, ) → nemo_microservices.types.evaluation.evaluation_job.EvaluationJob#

Start a new evaluation job by providing the necessary model and evaluation specifications.

A unique id will be created and provided in the response along with additional details.

Args: config: The evaluation configuration.

target: The evaluation target.

custom_fields: A set of custom fields that the user can define and use for various purposes.

description: The description of the entity.

namespace: The namespace of the entity. This can be missing for namespace entities or in deployments that don’t use namespaces.

output_files_url: The place where the output files, if any, should be written.

ownership: Information about ownership of an entity.

  If the entity is a namespace, the `access_policies` will typically apply to all
  entities inside the namespace.

project: The URN of the project associated with this entity.

result: The evaluation result URN.

status: Normalized statuses for all jobs.

  - **CREATED**: The job is created, but not yet scheduled.
  - **PENDING**: The job is waiting for resource allocation.
  - **RUNNING**: The job is currently running.
  - **CANCELLING**: The job is being cancelled at the user's request.
  - **CANCELLED**: The job has been cancelled by the user.
  - **CANCELLING**: The job is being cancelled at the user's request.
  - **FAILED**: The job failed to execute and terminated.
  - **COMPLETED**: The job has completed successfully.
  - **READY**: The job is ready to be used.
  - **UNKNOWN**: The job status is unknown.

status_details: Details about the status of the evaluation.

extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

delete( job_id: str, *, extra_headers: nemo_microservices._types.Headers | None = None, extra_query: nemo_microservices._types.Query | None = None, extra_body: nemo_microservices._types.Body | None = None, timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given, ) → None#

Delete Evaluation by Evaluation ID.

Args: extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

download_logs( job_id: str, *, extra_headers: nemo_microservices._types.Headers | None = None, extra_query: nemo_microservices._types.Query | None = None, extra_body: nemo_microservices._types.Body | None = None, timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given, ) → object#

Return the logs generated by the evaluation job.

The logs are returned as a ZIP file.

Args: extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

download_results( job_id: str, *, extra_headers: nemo_microservices._types.Headers | None = None, extra_query: nemo_microservices._types.Query | None = None, extra_body: nemo_microservices._types.Body | None = None, timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given, ) → nemo_microservices._response.BinaryAPIResponse#

Download the result directory containing configuration files, logs, and evaluation results for a specific evaluation run. The result directory is packaged and provided as a downloadable archive.

Args: extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

List available evaluation jobs.

Args: filter: Filter jobs on various criteria.

page: Page number.

page_size: Page size.

search: Search evaluation jobs using substring matching. You can combine multiple search fields and filters.

  For example:

  - `?search[status]=running`: searches all jobs with 'running' in the status.
  - `?search[config]=llama-benchmark`: searches all jobs with 'llama-benchmark' in
    the config field.
  - `?search[target]=model-accuracy`: searches all jobs with 'model-accuracy' in
    the target field.
  - `?search[status]=running&search[config]=llama-benchmark`: searches all jobs
    with 'running' in the status AND 'llama-benchmark' in the config field.
  - `?search[updated_at][start]=2024-01-01T00:00:00` finds all jobs updated on or
    after the start date
  - `?search[created_at][start]=2022-01-01&search[updated_at][end]=2024-01-01`
    finds all jobs created from start date up to and including end date

sort: The field to sort by. To sort in decreasing order, use - in front of the field name.

extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

results( job_id: str, *, extra_headers: nemo_microservices._types.Headers | None = None, extra_query: nemo_microservices._types.Query | None = None, extra_body: nemo_microservices._types.Body | None = None, timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given, ) → nemo_microservices.types.evaluation_result.EvaluationResult#

Return the results of the evaluation.

Args: extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

retrieve( job_id: str, *, extra_headers: nemo_microservices._types.Headers | None = None, extra_query: nemo_microservices._types.Query | None = None, extra_body: nemo_microservices._types.Body | None = None, timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given, ) → nemo_microservices.types.evaluation.evaluation_job.EvaluationJob#

Access detailed information about a specific evaluation.

Once an evaluation is finished, retrieve comprehensive results.

Args: extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

status( job_id: str, *, extra_headers: nemo_microservices._types.Headers | None = None, extra_query: nemo_microservices._types.Query | None = None, extra_body: nemo_microservices._types.Body | None = None, timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given, ) → nemo_microservices.types.evaluation_status_details.EvaluationStatusDetails#

Access detailed information about a specific evaluation.

Once an evaluation is finished, retrieve comprehensive results.

Args: extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

property with_raw_response: nemo_microservices.resources.evaluation.jobs.JobsResourceWithRawResponse#

This property can be used as a prefix for any HTTP method call to return the raw response object instead of the parsed content.

For more information, see https://docs.nvidia.com/nemo/microservices/latest/pysdk/index.html#accessing-raw-response-data-e-g-headers

property with_streaming_response: nemo_microservices.resources.evaluation.jobs.JobsResourceWithStreamingResponse#

An alternative to .with_raw_response that doesn’t eagerly read the response body.

For more information, see https://docs.nvidia.com/nemo/microservices/latest/pysdk/index.html#with_streaming_response

class nemo_microservices.resources.evaluation.jobs.JobsResourceWithRawResponse( jobs: nemo_microservices.resources.evaluation.jobs.JobsResource, )#: Initialization

class nemo_microservices.resources.evaluation.jobs.JobsResourceWithStreamingResponse( jobs: nemo_microservices.resources.evaluation.jobs.JobsResource, )#: Initialization

nemo_microservices.resources.evaluation.jobs#

Module Contents#

Classes#

API#

`nemo_microservices.resources.evaluation.jobs`#