nemo_gym.hf_utils

Module Contents

Functions

Name	Description
`check_jsonl_format`	Check for the presence of the expected keys in the dataset
`create_huggingface_client`	-
`download_hf_dataset_as_jsonl`	Download a HF dataset and save as JSONL.
`upload_jsonl_dataset`	-

API

nemo_gym.hf_utils.check_jsonl_format(
    file_path: str
) -> bool

Check for the presence of the expected keys in the dataset

nemo_gym.hf_utils.create_huggingface_client(
    token: str
) -> huggingface_hub.HfApi

nemo_gym.hf_utils.download_hf_dataset_as_jsonl(
    config: nemo_gym.config_types.DownloadJsonlDatasetHuggingFaceConfig
) -> None

Download a HF dataset and save as JSONL. If artifact_fpath is provided, downloads that specific file using hf_hub_download. Otherwise, uses datasets.load_dataset() to handle structured datasets.

nemo_gym.hf_utils.upload_jsonl_dataset(
    config: nemo_gym.config_types.UploadJsonlDatasetHuggingFaceConfig
) -> None