Download from Hugging Face

huggingface_hub.utils.HfHubHTTPError: 401 Client Error

Fix: Verify your token is valid. For gated datasets, accept the license on Hugging Face first.

huggingface_hub.utils.HfHubHTTPError: 404 Client Error

Fix: Check repo_id format is organization/dataset-name. Verify the repository exists and is public (or you have access).

ValueError: Either output_dirpath or output_fpath must be provided

Fix: Add +output_dirpath=./data/ or +output_fpath=./data/train.jsonl.

ValueError: Cannot specify both artifact_fpath and split

Fix: Use artifact_fpath for raw files OR split for structured datasets—not both.

Not recommended for shared systems:

$
ng_download_dataset_from_hf \
>
    +repo_id=my-org/private-dataset \
>
    +hf_token=hf_xxxxxxxxxxxxxxxxxxxxxxxxx \
>
    +output_dirpath=./data/

NeMo Gym can automatically download missing datasets during data preparation. Configure huggingface_identifier in your resources server config:

1
datasets:
2
  - name: train
3
    type: train
4
    jsonl_fpath: resources_servers/code_gen/data/train.jsonl
5
    huggingface_identifier:
6
      repo_id: nvidia/nemotron-RL-coding-competitive_coding
7
      artifact_fpath: opencodereasoning_filtered_25k_train.jsonl
8
    license: Apache 2.0

Run with download enabled:

$
config_paths="resources_servers/code_gen/configs/code_gen.yaml"
$
ng_prepare_data "+config_paths=[${config_paths}]" \
>
    +output_dirpath=./data/prepared \
>
    +mode=train_preparation \
>
    +should_download=true \
>
    +data_source=huggingface

If jsonl_fpath doesn’t exist locally, NeMo Gym downloads from huggingface_identifier before processing.

Downloads use Hugging Face’s cache at ~/.cache/huggingface/.

• Structured datasets: Reads from cache (fast), overwrites output file
• Raw files: Uses cached copy, then copies to output path

To force fresh download:

$
rm -rf ~/.cache/huggingface/hub/datasets--<org>--<dataset>