nemo_microservices.resources.datasets
#
Module Contents#
Classes#
API#
- class nemo_microservices.resources.datasets.AsyncDatasetsResource( )#
Bases:
nemo_microservices._resource.AsyncAPIResource
Initialization
- async create(
- *,
- files_url: str,
- custom_fields: Dict[str, object] | nemo_microservices._types.Omit = omit,
- description: str | nemo_microservices._types.Omit = omit,
- format: str | nemo_microservices._types.Omit = omit,
- hf_endpoint: str | nemo_microservices._types.Omit = omit,
- limit: int | nemo_microservices._types.Omit = omit,
- name: str | nemo_microservices._types.Omit = omit,
- namespace: str | nemo_microservices._types.Omit = omit,
- ownership: nemo_microservices.types.shared_params.ownership.Ownership | nemo_microservices._types.Omit = omit,
- project: str | nemo_microservices._types.Omit = omit,
- split: str | nemo_microservices._types.Omit = omit,
- extra_headers: nemo_microservices._types.Headers | None = None,
- extra_query: nemo_microservices._types.Query | None = None,
- extra_body: nemo_microservices._types.Body | None = None,
- timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given,
Create a new dataset.
Args: files_url: The location where the artifact files are stored. This can be a URL pointing to NDS, Hugging Face, S3, or any other accessible resource location.
custom_fields: A set of custom fields that the user can define and use for various purposes.
description: The description of the entity.
format: Specifies the dataset format, referring to the schema of the dataset rather than the file format. Examples include SQuAD, BEIR, etc.
hf_endpoint: For HuggingFace URLs, the endpoint that should be used. By default, this is set to the Data Store URL. For HuggingFace Hub, this should be set to “https://huggingface.co”.
limit: The maximum number of items to be used from the dataset.
name: The name of the entity. Must be unique inside the namespace. If not specified, it will be the same as the automatically generated id.
namespace: The namespace of the entity. This can be missing for namespace entities or in deployments that don’t use namespaces.
ownership: Information about ownership of an entity.
If the entity is a namespace, the `access_policies` will typically apply to all entities inside the namespace.
project: The URN of the project associated with this entity.
split: The split of the dataset. Examples include train, validation, test, etc.
extra_headers: Send extra headers
extra_query: Add additional query parameters to the request
extra_body: Add additional JSON properties to the request
timeout: Override the client-level default timeout for this request, in seconds
- async delete(
- dataset_name: str,
- *,
- namespace: str,
- extra_headers: nemo_microservices._types.Headers | None = None,
- extra_query: nemo_microservices._types.Query | None = None,
- extra_body: nemo_microservices._types.Body | None = None,
- timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given,
Delete Dataset
Args: extra_headers: Send extra headers
extra_query: Add additional query parameters to the request
extra_body: Add additional JSON properties to the request
timeout: Override the client-level default timeout for this request, in seconds
- list(
- *,
- filter: nemo_microservices.types.dataset_filter_param.DatasetFilterParam | nemo_microservices._types.Omit = omit,
- page: int | nemo_microservices._types.Omit = omit,
- page_size: int | nemo_microservices._types.Omit = omit,
- search: nemo_microservices.types.dataset_search_param.DatasetSearchParam | nemo_microservices._types.Omit = omit,
- sort: nemo_microservices.types.dataset_sort_field.DatasetSortField | nemo_microservices._types.Omit = omit,
- extra_headers: nemo_microservices._types.Headers | None = None,
- extra_query: nemo_microservices._types.Query | None = None,
- extra_body: nemo_microservices._types.Body | None = None,
- timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given,
List all datasets.
Args: filter: Filter configs on various criteria.
page: Page number.
page_size: Page size.
search: Search datasets using substring matching. You can combine multiple search fields and filters.
For example: - `?search[name]=imagenet`: searches all datasets with 'imagenet' in the name. - `?search[format]=csv`: searches all datasets with 'csv' in the format. - `?search[split]=train`: searches all datasets with 'train' in the split. - `?search[namespace]=research`: searches all datasets with 'research' in the namespace. - `?search[name]=imagenet&search[split]=validation`: searches all datasets with 'imagenet' in the name AND 'validation' in the split. - `?search[name]=imagenet&search[name]=coco`: searches all datasets with 'imagenet' OR 'coco' in the name. - `?search[updated_at][start]=2024-01-01T00:00:00` finds all datasets updated on or after the start date - `?search[created_at][start]=2022-01-01&search[updated_at][end]=2024-01-01` finds all datasets created from start date up to and including end date
sort: The field to sort by. To sort in decreasing order, use
-
in front of the field name.extra_headers: Send extra headers
extra_query: Add additional query parameters to the request
extra_body: Add additional JSON properties to the request
timeout: Override the client-level default timeout for this request, in seconds
- async retrieve(
- dataset_name: str,
- *,
- namespace: str,
- extra_headers: nemo_microservices._types.Headers | None = None,
- extra_query: nemo_microservices._types.Query | None = None,
- extra_body: nemo_microservices._types.Body | None = None,
- timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given,
Get Dataset
Args: extra_headers: Send extra headers
extra_query: Add additional query parameters to the request
extra_body: Add additional JSON properties to the request
timeout: Override the client-level default timeout for this request, in seconds
- async update(
- dataset_name: str,
- *,
- namespace: str,
- custom_fields: Dict[str, object] | nemo_microservices._types.Omit = omit,
- description: str | nemo_microservices._types.Omit = omit,
- files_url: str | nemo_microservices._types.Omit = omit,
- format: str | nemo_microservices._types.Omit = omit,
- hf_endpoint: str | nemo_microservices._types.Omit = omit,
- limit: int | nemo_microservices._types.Omit = omit,
- ownership: nemo_microservices.types.shared_params.ownership.Ownership | nemo_microservices._types.Omit = omit,
- project: str | nemo_microservices._types.Omit = omit,
- split: str | nemo_microservices._types.Omit = omit,
- extra_headers: nemo_microservices._types.Headers | None = None,
- extra_query: nemo_microservices._types.Query | None = None,
- extra_body: nemo_microservices._types.Body | None = None,
- timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given,
Update dataset metadata.
If the request body has an empty field, keep the old value.
Args: custom_fields: A set of custom fields that the user can define and use for various purposes.
description: The description of the entity.
files_url: The location where the artifact files are stored. This can be a URL pointing to NDS, Hugging Face, S3, or any other accessible resource location.
format: Specifies the dataset format, referring to the schema of the dataset rather than the file format. Examples include SQuAD, BEIR, etc.
hf_endpoint: For HuggingFace URLs, the endpoint that should be used. By default, this is set to the Data Store URL. For HuggingFace Hub, this should be set to “https://huggingface.co”.
limit: The maximum number of items to be used from the dataset.
ownership: Information about ownership of an entity.
If the entity is a namespace, the `access_policies` will typically apply to all entities inside the namespace.
project: The URN of the project associated with this entity.
split: The split of the dataset. Examples include train, validation, test, etc.
extra_headers: Send extra headers
extra_query: Add additional query parameters to the request
extra_body: Add additional JSON properties to the request
timeout: Override the client-level default timeout for this request, in seconds
- property with_raw_response: nemo_microservices.resources.datasets.AsyncDatasetsResourceWithRawResponse#
This property can be used as a prefix for any HTTP method call to return the raw response object instead of the parsed content.
For more information, see https://docs.nvidia.com/nemo/microservices/latest/pysdk/index.html#accessing-raw-response-data-e-g-headers
- property with_streaming_response: nemo_microservices.resources.datasets.AsyncDatasetsResourceWithStreamingResponse#
An alternative to
.with_raw_response
that doesn’t eagerly read the response body.For more information, see https://docs.nvidia.com/nemo/microservices/latest/pysdk/index.html#with_streaming_response
- class nemo_microservices.resources.datasets.AsyncDatasetsResourceWithRawResponse( )#
Initialization
- class nemo_microservices.resources.datasets.AsyncDatasetsResourceWithStreamingResponse( )#
Initialization
- class nemo_microservices.resources.datasets.DatasetsResource(client: nemo_microservices._client.NeMoMicroservices)#
Bases:
nemo_microservices._resource.SyncAPIResource
Initialization
- create(
- *,
- files_url: str,
- custom_fields: Dict[str, object] | nemo_microservices._types.Omit = omit,
- description: str | nemo_microservices._types.Omit = omit,
- format: str | nemo_microservices._types.Omit = omit,
- hf_endpoint: str | nemo_microservices._types.Omit = omit,
- limit: int | nemo_microservices._types.Omit = omit,
- name: str | nemo_microservices._types.Omit = omit,
- namespace: str | nemo_microservices._types.Omit = omit,
- ownership: nemo_microservices.types.shared_params.ownership.Ownership | nemo_microservices._types.Omit = omit,
- project: str | nemo_microservices._types.Omit = omit,
- split: str | nemo_microservices._types.Omit = omit,
- extra_headers: nemo_microservices._types.Headers | None = None,
- extra_query: nemo_microservices._types.Query | None = None,
- extra_body: nemo_microservices._types.Body | None = None,
- timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given,
Create a new dataset.
Args: files_url: The location where the artifact files are stored. This can be a URL pointing to NDS, Hugging Face, S3, or any other accessible resource location.
custom_fields: A set of custom fields that the user can define and use for various purposes.
description: The description of the entity.
format: Specifies the dataset format, referring to the schema of the dataset rather than the file format. Examples include SQuAD, BEIR, etc.
hf_endpoint: For HuggingFace URLs, the endpoint that should be used. By default, this is set to the Data Store URL. For HuggingFace Hub, this should be set to “https://huggingface.co”.
limit: The maximum number of items to be used from the dataset.
name: The name of the entity. Must be unique inside the namespace. If not specified, it will be the same as the automatically generated id.
namespace: The namespace of the entity. This can be missing for namespace entities or in deployments that don’t use namespaces.
ownership: Information about ownership of an entity.
If the entity is a namespace, the `access_policies` will typically apply to all entities inside the namespace.
project: The URN of the project associated with this entity.
split: The split of the dataset. Examples include train, validation, test, etc.
extra_headers: Send extra headers
extra_query: Add additional query parameters to the request
extra_body: Add additional JSON properties to the request
timeout: Override the client-level default timeout for this request, in seconds
- delete(
- dataset_name: str,
- *,
- namespace: str,
- extra_headers: nemo_microservices._types.Headers | None = None,
- extra_query: nemo_microservices._types.Query | None = None,
- extra_body: nemo_microservices._types.Body | None = None,
- timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given,
Delete Dataset
Args: extra_headers: Send extra headers
extra_query: Add additional query parameters to the request
extra_body: Add additional JSON properties to the request
timeout: Override the client-level default timeout for this request, in seconds
- list(
- *,
- filter: nemo_microservices.types.dataset_filter_param.DatasetFilterParam | nemo_microservices._types.Omit = omit,
- page: int | nemo_microservices._types.Omit = omit,
- page_size: int | nemo_microservices._types.Omit = omit,
- search: nemo_microservices.types.dataset_search_param.DatasetSearchParam | nemo_microservices._types.Omit = omit,
- sort: nemo_microservices.types.dataset_sort_field.DatasetSortField | nemo_microservices._types.Omit = omit,
- extra_headers: nemo_microservices._types.Headers | None = None,
- extra_query: nemo_microservices._types.Query | None = None,
- extra_body: nemo_microservices._types.Body | None = None,
- timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given,
List all datasets.
Args: filter: Filter configs on various criteria.
page: Page number.
page_size: Page size.
search: Search datasets using substring matching. You can combine multiple search fields and filters.
For example: - `?search[name]=imagenet`: searches all datasets with 'imagenet' in the name. - `?search[format]=csv`: searches all datasets with 'csv' in the format. - `?search[split]=train`: searches all datasets with 'train' in the split. - `?search[namespace]=research`: searches all datasets with 'research' in the namespace. - `?search[name]=imagenet&search[split]=validation`: searches all datasets with 'imagenet' in the name AND 'validation' in the split. - `?search[name]=imagenet&search[name]=coco`: searches all datasets with 'imagenet' OR 'coco' in the name. - `?search[updated_at][start]=2024-01-01T00:00:00` finds all datasets updated on or after the start date - `?search[created_at][start]=2022-01-01&search[updated_at][end]=2024-01-01` finds all datasets created from start date up to and including end date
sort: The field to sort by. To sort in decreasing order, use
-
in front of the field name.extra_headers: Send extra headers
extra_query: Add additional query parameters to the request
extra_body: Add additional JSON properties to the request
timeout: Override the client-level default timeout for this request, in seconds
- retrieve(
- dataset_name: str,
- *,
- namespace: str,
- extra_headers: nemo_microservices._types.Headers | None = None,
- extra_query: nemo_microservices._types.Query | None = None,
- extra_body: nemo_microservices._types.Body | None = None,
- timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given,
Get Dataset
Args: extra_headers: Send extra headers
extra_query: Add additional query parameters to the request
extra_body: Add additional JSON properties to the request
timeout: Override the client-level default timeout for this request, in seconds
- update(
- dataset_name: str,
- *,
- namespace: str,
- custom_fields: Dict[str, object] | nemo_microservices._types.Omit = omit,
- description: str | nemo_microservices._types.Omit = omit,
- files_url: str | nemo_microservices._types.Omit = omit,
- format: str | nemo_microservices._types.Omit = omit,
- hf_endpoint: str | nemo_microservices._types.Omit = omit,
- limit: int | nemo_microservices._types.Omit = omit,
- ownership: nemo_microservices.types.shared_params.ownership.Ownership | nemo_microservices._types.Omit = omit,
- project: str | nemo_microservices._types.Omit = omit,
- split: str | nemo_microservices._types.Omit = omit,
- extra_headers: nemo_microservices._types.Headers | None = None,
- extra_query: nemo_microservices._types.Query | None = None,
- extra_body: nemo_microservices._types.Body | None = None,
- timeout: float | httpx.Timeout | None | nemo_microservices._types.NotGiven = not_given,
Update dataset metadata.
If the request body has an empty field, keep the old value.
Args: custom_fields: A set of custom fields that the user can define and use for various purposes.
description: The description of the entity.
files_url: The location where the artifact files are stored. This can be a URL pointing to NDS, Hugging Face, S3, or any other accessible resource location.
format: Specifies the dataset format, referring to the schema of the dataset rather than the file format. Examples include SQuAD, BEIR, etc.
hf_endpoint: For HuggingFace URLs, the endpoint that should be used. By default, this is set to the Data Store URL. For HuggingFace Hub, this should be set to “https://huggingface.co”.
limit: The maximum number of items to be used from the dataset.
ownership: Information about ownership of an entity.
If the entity is a namespace, the `access_policies` will typically apply to all entities inside the namespace.
project: The URN of the project associated with this entity.
split: The split of the dataset. Examples include train, validation, test, etc.
extra_headers: Send extra headers
extra_query: Add additional query parameters to the request
extra_body: Add additional JSON properties to the request
timeout: Override the client-level default timeout for this request, in seconds
- property with_raw_response: nemo_microservices.resources.datasets.DatasetsResourceWithRawResponse#
This property can be used as a prefix for any HTTP method call to return the raw response object instead of the parsed content.
For more information, see https://docs.nvidia.com/nemo/microservices/latest/pysdk/index.html#accessing-raw-response-data-e-g-headers
- property with_streaming_response: nemo_microservices.resources.datasets.DatasetsResourceWithStreamingResponse#
An alternative to
.with_raw_response
that doesn’t eagerly read the response body.For more information, see https://docs.nvidia.com/nemo/microservices/latest/pysdk/index.html#with_streaming_response
- class nemo_microservices.resources.datasets.DatasetsResourceWithRawResponse( )#
Initialization
- class nemo_microservices.resources.datasets.DatasetsResourceWithStreamingResponse( )#
Initialization