`nemo_microservices.data_designer.config.datastore`#

Module Contents#

Classes#

DatastoreSettings

Configuration for interacting with a datastore.

Functions#

`fetch_seed_dataset_column_names`
`fetch_seed_dataset_column_names_from_datastore`
`fetch_seed_dataset_column_names_from_local_file`
`get_file_column_names`	Get column names from a dataset file.
`resolve_datastore_settings`
`upload_to_hf_hub`

Data#

logger

API#

class nemo_microservices.data_designer.config.datastore.DatastoreSettings(/, **data: typing.Any)#

Bases: pydantic.BaseModel

Configuration for interacting with a datastore.

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

endpoint: str#: ‘Field(…)’

token: Optional[str]#: ‘Field(…)’

nemo_microservices.data_designer.config.datastore.fetch_seed_dataset_column_names( seed_dataset_reference: nemo_microservices.data_designer.config.seed.SeedDatasetReference, ) → list[str]#

nemo_microservices.data_designer.config.datastore.fetch_seed_dataset_column_names_from_datastore( repo_id: str, filename: str, datastore_settings: Optional[Union[nemo_microservices.data_designer.config.datastore.DatastoreSettings, dict]] = None, ) → list[str]#

nemo_microservices.data_designer.config.datastore.fetch_seed_dataset_column_names_from_local_file( dataset_path: str | pathlib.Path, ) → list[str]#

nemo_microservices.data_designer.config.datastore.get_file_column_names( file_reference: Union[str, pathlib.Path, huggingface_hub.HfFileSystem], file_type: str, ) → list[str]#

Get column names from a dataset file.

Args: file_reference: Path to the dataset file, or an HfFileSystem object. file_type: Type of the dataset file. Must be one of: ‘parquet’, ‘json’, ‘jsonl’, ‘csv’.

Raises: InvalidFilePathError: If the file type is not supported.

Returns: List of column names.

nemo_microservices.data_designer.config.datastore.logger#: ‘getLogger(…)’

nemo_microservices.data_designer.config.datastore.resolve_datastore_settings( datastore_settings: nemo_microservices.data_designer.config.datastore.DatastoreSettings | dict | None, ) → nemo_microservices.data_designer.config.datastore.DatastoreSettings#

nemo_microservices.data_designer.config.datastore.upload_to_hf_hub(

dataset_path: Union[str, pathlib.Path],

filename: str,

repo_id: str,

datastore_settings: nemo_microservices.data_designer.config.datastore.DatastoreSettings,

**kwargs,

) → str#

nemo_microservices.data_designer.config.datastore#