nemo_gym.hf_utils

View as Markdown

Module Contents

Functions

NameDescription
check_jsonl_formatCheck for the presence of the expected keys in the dataset
create_huggingface_client-
download_hf_dataset_as_jsonlDownload a HF dataset and save as JSONL.
upload_jsonl_dataset-

API

nemo_gym.hf_utils.check_jsonl_format(
file_path: str
) -> bool

Check for the presence of the expected keys in the dataset

nemo_gym.hf_utils.create_huggingface_client(
token: str
) -> huggingface_hub.HfApi
nemo_gym.hf_utils.download_hf_dataset_as_jsonl(
config: nemo_gym.config_types.DownloadJsonlDatasetHuggingFaceConfig
) -> None

Download a HF dataset and save as JSONL. If artifact_fpath is provided, downloads that specific file using hf_hub_download. Otherwise, uses datasets.load_dataset() to handle structured datasets.

nemo_gym.hf_utils.upload_jsonl_dataset(
config: nemo_gym.config_types.UploadJsonlDatasetHuggingFaceConfig
) -> None