nemo_microservices.resources.datasets#

Module Contents#

Classes#

API#

class nemo_microservices.resources.datasets.AsyncDatasetsResource(
client: nemo_microservices._client.AsyncNeMoMicroservices,
)#

Bases: nemo_microservices._resource.AsyncAPIResource

Initialization

async create(
*,
files_url: str,
custom_fields: Dict[str, object] | nemo_microservices._types.Omit = omit,
description: str | nemo_microservices._types.Omit = omit,
format: str | nemo_microservices._types.Omit = omit,
hf_endpoint: str | nemo_microservices._types.Omit = omit,
limit: int | nemo_microservices._types.Omit = omit,
name: str | nemo_microservices._types.Omit = omit,
namespace: str | nemo_microservices._types.Omit = omit,
ownership: nemo_microservices.types.shared_params.ownership.Ownership | nemo_microservices._types.Omit = omit,
project: str | nemo_microservices._types.Omit = omit,
split: str | nemo_microservices._types.Omit = omit,
extra_headers: nemo_microservices._types.Headers | None = None,
extra_query: nemo_microservices._types.Query | None = None,
extra_body: nemo_microservices._types.Body | None = None,
timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given,
) nemo_microservices.types.dataset.Dataset#

Create a new dataset.

Args: files_url: The location where the artifact files are stored. This can be a URL pointing to NDS, Hugging Face, S3, or any other accessible resource location.

custom_fields: A set of custom fields that the user can define and use for various purposes.

description: The description of the entity.

format: Specifies the dataset format, referring to the schema of the dataset rather than the file format. Examples include SQuAD, BEIR, etc.

hf_endpoint: For HuggingFace URLs, the endpoint that should be used. By default, this is set to the Data Store URL. For HuggingFace Hub, this should be set to “https://huggingface.co”.

limit: The maximum number of items to be used from the dataset.

name: The name of the entity. Must be unique inside the namespace. If not specified, it will be the same as the automatically generated id.

namespace: The namespace of the entity. This can be missing for namespace entities or in deployments that don’t use namespaces.

ownership: Information about ownership of an entity.

  If the entity is a namespace, the `access_policies` will typically apply to all
  entities inside the namespace.

project: The URN of the project associated with this entity.

split: The split of the dataset. Examples include train, validation, test, etc.

extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

async delete(
dataset_name: str,
*,
namespace: str,
extra_headers: nemo_microservices._types.Headers | None = None,
extra_query: nemo_microservices._types.Query | None = None,
extra_body: nemo_microservices._types.Body | None = None,
timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given,
) nemo_microservices.types.shared.delete_response.DeleteResponse#

Delete Dataset

Args: extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

list(
*,
filter: nemo_microservices.types.dataset_filter_param.DatasetFilterParam | nemo_microservices._types.Omit = omit,
page: int | nemo_microservices._types.Omit = omit,
page_size: int | nemo_microservices._types.Omit = omit,
search: nemo_microservices.types.dataset_search_param.DatasetSearchParam | nemo_microservices._types.Omit = omit,
sort: nemo_microservices.types.dataset_sort_field.DatasetSortField | nemo_microservices._types.Omit = omit,
extra_headers: nemo_microservices._types.Headers | None = None,
extra_query: nemo_microservices._types.Query | None = None,
extra_body: nemo_microservices._types.Body | None = None,
timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given,
) nemo_microservices._base_client.AsyncPaginator[nemo_microservices.types.dataset.Dataset, nemo_microservices.pagination.AsyncDefaultPagination[nemo_microservices.types.dataset.Dataset]]#

List all datasets.

Args: filter: Filter configs on various criteria.

page: Page number.

page_size: Page size.

search: Search datasets using substring matching. You can combine multiple search fields and filters.

  For example:

  - `?search[name]=imagenet`: searches all datasets with 'imagenet' in the name.
  - `?search[format]=csv`: searches all datasets with 'csv' in the format.
  - `?search[split]=train`: searches all datasets with 'train' in the split.
  - `?search[namespace]=research`: searches all datasets with 'research' in the
    namespace.
  - `?search[name]=imagenet&search[split]=validation`: searches all datasets with
    'imagenet' in the name AND 'validation' in the split.
  - `?search[name]=imagenet&search[name]=coco`: searches all datasets with
    'imagenet' OR 'coco' in the name.
  - `?search[updated_at][start]=2024-01-01T00:00:00` finds all datasets updated on
    or after the start date
  - `?search[created_at][start]=2022-01-01&search[updated_at][end]=2024-01-01`
    finds all datasets created from start date up to and including end date

sort: The field to sort by. To sort in decreasing order, use - in front of the field name.

extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

async retrieve(
dataset_name: str,
*,
namespace: str,
extra_headers: nemo_microservices._types.Headers | None = None,
extra_query: nemo_microservices._types.Query | None = None,
extra_body: nemo_microservices._types.Body | None = None,
timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given,
) nemo_microservices.types.dataset.Dataset#

Get Dataset

Args: extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

async update(
dataset_name: str,
*,
namespace: str,
custom_fields: Dict[str, object] | nemo_microservices._types.Omit = omit,
description: str | nemo_microservices._types.Omit = omit,
files_url: str | nemo_microservices._types.Omit = omit,
format: str | nemo_microservices._types.Omit = omit,
hf_endpoint: str | nemo_microservices._types.Omit = omit,
limit: int | nemo_microservices._types.Omit = omit,
ownership: nemo_microservices.types.shared_params.ownership.Ownership | nemo_microservices._types.Omit = omit,
project: str | nemo_microservices._types.Omit = omit,
split: str | nemo_microservices._types.Omit = omit,
extra_headers: nemo_microservices._types.Headers | None = None,
extra_query: nemo_microservices._types.Query | None = None,
extra_body: nemo_microservices._types.Body | None = None,
timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given,
) nemo_microservices.types.dataset.Dataset#

Update dataset metadata.

If the request body has an empty field, keep the old value.

Args: custom_fields: A set of custom fields that the user can define and use for various purposes.

description: The description of the entity.

files_url: The location where the artifact files are stored. This can be a URL pointing to NDS, Hugging Face, S3, or any other accessible resource location.

format: Specifies the dataset format, referring to the schema of the dataset rather than the file format. Examples include SQuAD, BEIR, etc.

hf_endpoint: For HuggingFace URLs, the endpoint that should be used. By default, this is set to the Data Store URL. For HuggingFace Hub, this should be set to “https://huggingface.co”.

limit: The maximum number of items to be used from the dataset.

ownership: Information about ownership of an entity.

  If the entity is a namespace, the `access_policies` will typically apply to all
  entities inside the namespace.

project: The URN of the project associated with this entity.

split: The split of the dataset. Examples include train, validation, test, etc.

extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

property with_raw_response: nemo_microservices.resources.datasets.AsyncDatasetsResourceWithRawResponse#

This property can be used as a prefix for any HTTP method call to return the raw response object instead of the parsed content.

For more information, see https://docs.nvidia.com/nemo/microservices/latest/pysdk/index.html#accessing-raw-response-data-e-g-headers

property with_streaming_response: nemo_microservices.resources.datasets.AsyncDatasetsResourceWithStreamingResponse#

An alternative to .with_raw_response that doesn’t eagerly read the response body.

For more information, see https://docs.nvidia.com/nemo/microservices/latest/pysdk/index.html#with_streaming_response

class nemo_microservices.resources.datasets.AsyncDatasetsResourceWithRawResponse(
datasets: nemo_microservices.resources.datasets.AsyncDatasetsResource,
)#

Initialization

class nemo_microservices.resources.datasets.AsyncDatasetsResourceWithStreamingResponse(
datasets: nemo_microservices.resources.datasets.AsyncDatasetsResource,
)#

Initialization

class nemo_microservices.resources.datasets.DatasetsResource(client: nemo_microservices._client.NeMoMicroservices)#

Bases: nemo_microservices._resource.SyncAPIResource

Initialization

create(
*,
files_url: str,
custom_fields: Dict[str, object] | nemo_microservices._types.Omit = omit,
description: str | nemo_microservices._types.Omit = omit,
format: str | nemo_microservices._types.Omit = omit,
hf_endpoint: str | nemo_microservices._types.Omit = omit,
limit: int | nemo_microservices._types.Omit = omit,
name: str | nemo_microservices._types.Omit = omit,
namespace: str | nemo_microservices._types.Omit = omit,
ownership: nemo_microservices.types.shared_params.ownership.Ownership | nemo_microservices._types.Omit = omit,
project: str | nemo_microservices._types.Omit = omit,
split: str | nemo_microservices._types.Omit = omit,
extra_headers: nemo_microservices._types.Headers | None = None,
extra_query: nemo_microservices._types.Query | None = None,
extra_body: nemo_microservices._types.Body | None = None,
timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given,
) nemo_microservices.types.dataset.Dataset#

Create a new dataset.

Args: files_url: The location where the artifact files are stored. This can be a URL pointing to NDS, Hugging Face, S3, or any other accessible resource location.

custom_fields: A set of custom fields that the user can define and use for various purposes.

description: The description of the entity.

format: Specifies the dataset format, referring to the schema of the dataset rather than the file format. Examples include SQuAD, BEIR, etc.

hf_endpoint: For HuggingFace URLs, the endpoint that should be used. By default, this is set to the Data Store URL. For HuggingFace Hub, this should be set to “https://huggingface.co”.

limit: The maximum number of items to be used from the dataset.

name: The name of the entity. Must be unique inside the namespace. If not specified, it will be the same as the automatically generated id.

namespace: The namespace of the entity. This can be missing for namespace entities or in deployments that don’t use namespaces.

ownership: Information about ownership of an entity.

  If the entity is a namespace, the `access_policies` will typically apply to all
  entities inside the namespace.

project: The URN of the project associated with this entity.

split: The split of the dataset. Examples include train, validation, test, etc.

extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

delete(
dataset_name: str,
*,
namespace: str,
extra_headers: nemo_microservices._types.Headers | None = None,
extra_query: nemo_microservices._types.Query | None = None,
extra_body: nemo_microservices._types.Body | None = None,
timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given,
) nemo_microservices.types.shared.delete_response.DeleteResponse#

Delete Dataset

Args: extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

list(
*,
filter: nemo_microservices.types.dataset_filter_param.DatasetFilterParam | nemo_microservices._types.Omit = omit,
page: int | nemo_microservices._types.Omit = omit,
page_size: int | nemo_microservices._types.Omit = omit,
search: nemo_microservices.types.dataset_search_param.DatasetSearchParam | nemo_microservices._types.Omit = omit,
sort: nemo_microservices.types.dataset_sort_field.DatasetSortField | nemo_microservices._types.Omit = omit,
extra_headers: nemo_microservices._types.Headers | None = None,
extra_query: nemo_microservices._types.Query | None = None,
extra_body: nemo_microservices._types.Body | None = None,
timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given,
) nemo_microservices.pagination.SyncDefaultPagination[nemo_microservices.types.dataset.Dataset]#

List all datasets.

Args: filter: Filter configs on various criteria.

page: Page number.

page_size: Page size.

search: Search datasets using substring matching. You can combine multiple search fields and filters.

  For example:

  - `?search[name]=imagenet`: searches all datasets with 'imagenet' in the name.
  - `?search[format]=csv`: searches all datasets with 'csv' in the format.
  - `?search[split]=train`: searches all datasets with 'train' in the split.
  - `?search[namespace]=research`: searches all datasets with 'research' in the
    namespace.
  - `?search[name]=imagenet&search[split]=validation`: searches all datasets with
    'imagenet' in the name AND 'validation' in the split.
  - `?search[name]=imagenet&search[name]=coco`: searches all datasets with
    'imagenet' OR 'coco' in the name.
  - `?search[updated_at][start]=2024-01-01T00:00:00` finds all datasets updated on
    or after the start date
  - `?search[created_at][start]=2022-01-01&search[updated_at][end]=2024-01-01`
    finds all datasets created from start date up to and including end date

sort: The field to sort by. To sort in decreasing order, use - in front of the field name.

extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

retrieve(
dataset_name: str,
*,
namespace: str,
extra_headers: nemo_microservices._types.Headers | None = None,
extra_query: nemo_microservices._types.Query | None = None,
extra_body: nemo_microservices._types.Body | None = None,
timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given,
) nemo_microservices.types.dataset.Dataset#

Get Dataset

Args: extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

update(
dataset_name: str,
*,
namespace: str,
custom_fields: Dict[str, object] | nemo_microservices._types.Omit = omit,
description: str | nemo_microservices._types.Omit = omit,
files_url: str | nemo_microservices._types.Omit = omit,
format: str | nemo_microservices._types.Omit = omit,
hf_endpoint: str | nemo_microservices._types.Omit = omit,
limit: int | nemo_microservices._types.Omit = omit,
ownership: nemo_microservices.types.shared_params.ownership.Ownership | nemo_microservices._types.Omit = omit,
project: str | nemo_microservices._types.Omit = omit,
split: str | nemo_microservices._types.Omit = omit,
extra_headers: nemo_microservices._types.Headers | None = None,
extra_query: nemo_microservices._types.Query | None = None,
extra_body: nemo_microservices._types.Body | None = None,
timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given,
) nemo_microservices.types.dataset.Dataset#

Update dataset metadata.

If the request body has an empty field, keep the old value.

Args: custom_fields: A set of custom fields that the user can define and use for various purposes.

description: The description of the entity.

files_url: The location where the artifact files are stored. This can be a URL pointing to NDS, Hugging Face, S3, or any other accessible resource location.

format: Specifies the dataset format, referring to the schema of the dataset rather than the file format. Examples include SQuAD, BEIR, etc.

hf_endpoint: For HuggingFace URLs, the endpoint that should be used. By default, this is set to the Data Store URL. For HuggingFace Hub, this should be set to “https://huggingface.co”.

limit: The maximum number of items to be used from the dataset.

ownership: Information about ownership of an entity.

  If the entity is a namespace, the `access_policies` will typically apply to all
  entities inside the namespace.

project: The URN of the project associated with this entity.

split: The split of the dataset. Examples include train, validation, test, etc.

extra_headers: Send extra headers

extra_query: Add additional query parameters to the request

extra_body: Add additional JSON properties to the request

timeout: Override the client-level default timeout for this request, in seconds

property with_raw_response: nemo_microservices.resources.datasets.DatasetsResourceWithRawResponse#

This property can be used as a prefix for any HTTP method call to return the raw response object instead of the parsed content.

For more information, see https://docs.nvidia.com/nemo/microservices/latest/pysdk/index.html#accessing-raw-response-data-e-g-headers

property with_streaming_response: nemo_microservices.resources.datasets.DatasetsResourceWithStreamingResponse#

An alternative to .with_raw_response that doesn’t eagerly read the response body.

For more information, see https://docs.nvidia.com/nemo/microservices/latest/pysdk/index.html#with_streaming_response

class nemo_microservices.resources.datasets.DatasetsResourceWithRawResponse(
datasets: nemo_microservices.resources.datasets.DatasetsResource,
)#

Initialization

class nemo_microservices.resources.datasets.DatasetsResourceWithStreamingResponse(
datasets: nemo_microservices.resources.datasets.DatasetsResource,
)#

Initialization