> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# Download from Hugging Face

Download JSONL datasets from Hugging Face Hub for NeMo Gym training.

**Goal**: Download a dataset from Hugging Face Hub in JSONL format for training.

**Prerequisites**: NeMo Gym installed ([Installation](/get-started/installation))

---

## Quick Start

```bash
gym dataset download \
    --repo-id nvidia/Nemotron-RL-math-OpenMathReasoning \
    --split train \
    --output ./data/train.jsonl
```

```text
[Nemo-Gym] - Downloaded train split to: ./data/train.jsonl
```

> **Note**
>
> NeMo Gym uses [Hydra](https://hydra.cc/) for configuration. Arguments use `+key=value` syntax.

---

## Options

| Option           | Description                                                                                     |
| ---------------- | ----------------------------------------------------------------------------------------------- |
| `repo_id`        | **Required.** Hugging Face repository (e.g., `nvidia/Nemotron-RL-math-OpenMathReasoning`)       |
| `output_dirpath` | Output directory. Files named `{split}.jsonl`. **Use this OR `output_fpath`.**                  |
| `output_fpath`   | Exact output file path. Requires `split` or `artifact_fpath`. **Use this OR `output_dirpath`.** |
| `artifact_fpath` | Download a specific file from the repo (raw file mode)                                          |
| `split`          | Dataset split: `train`, `validation`, or `test`. Omit to download all.                          |
| `hf_token`       | Authentication token for private/gated repositories                                             |

---

## Download Methods

#### Structured Dataset (Recommended)

Downloads using the `datasets` library and converts to JSONL.

**Use when**: Repository uses Hugging Face's standard dataset format.

**All splits**:

```bash
gym dataset download \
    --repo-id nvidia/Nemotron-RL-knowledge-mcqa \
    --output-dir ./data/
```

```text
[Nemo-Gym] - Downloaded train split to: ./data/train.jsonl
[Nemo-Gym] - Downloaded validation split to: ./data/validation.jsonl
```

**Single split**:

```bash
gym dataset download \
    --repo-id SWE-Gym/SWE-Gym \
    --split train \
    --output ./data/train.jsonl
```

#### Raw File

Downloads a specific file directly without conversion.

**Use when**: Repository contains pre-formatted JSONL files.

```bash
gym dataset download \
    --repo-id nvidia/nemotron-RL-coding-competitive_coding \
    --artifact opencodereasoning_filtered_25k_train.jsonl \
    --output ./data/train.jsonl
```

```text
[Nemo-Gym] - Downloaded opencodereasoning_filtered_25k_train.jsonl to: ./data/train.jsonl
```

#### Python Script

Downloads using the `datasets` library directly with streaming support.

**Use when**: You need custom preprocessing, streaming for large datasets, or specific split handling.

```python
import json
from datasets import load_dataset

output_file = "train.jsonl"
dataset_name = "nvidia/OpenMathInstruct-2"
split_name = "train_1M"  # Check dataset page for available splits

with open(output_file, "w", encoding="utf-8") as f:
    for line in load_dataset(dataset_name, split=split_name, streaming=True):
        f.write(json.dumps(line) + "\n")
```

Run the script:

```bash
uv run download.py
```

Verify the download:

```bash
wc -l train.jsonl
# Expected: 1000000 train.jsonl
```

**Streaming benefits**:

* Memory-efficient for large datasets (millions of rows)
* Progress visible during download

> **Note**
>
> For gated or private datasets, authenticate first:
>
> ```bash
> export HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxx
> ```
>
> Or use `huggingface-cli login` before running the script.

---

## NVIDIA Datasets

Ready-to-use datasets for common training tasks:

| Dataset             | Repository                                                    | Domain      |
| ------------------- | ------------------------------------------------------------- | ----------- |
| OpenMathReasoning   | `nvidia/Nemotron-RL-math-OpenMathReasoning`                   | Math        |
| Competitive Coding  | `nvidia/nemotron-RL-coding-competitive_coding`                | Code        |
| Workplace Assistant | `nvidia/Nemotron-RL-agent-workplace_assistant`                | Agent       |
| Structured Outputs  | `nvidia/Nemotron-RL-instruction_following-structured_outputs` | Instruction |
| MCQA                | `nvidia/Nemotron-RL-knowledge-mcqa`                           | Knowledge   |

---

## Troubleshooting

#### Authentication Failed (401)

```text
huggingface_hub.utils.HfHubHTTPError: 401 Client Error
```

**Fix**: Verify your token is valid. For gated datasets, accept the license on Hugging Face first.

#### Repository Not Found (404)

```text
huggingface_hub.utils.HfHubHTTPError: 404 Client Error
```

**Fix**: Check `repo_id` format is `organization/dataset-name`. Verify the repository exists and is public (or you have access).

#### Validation Error: Output Path

```text
ValueError: Either output_dirpath or output_fpath must be provided
```

**Fix**: Add `+output_dirpath=./data/` or `+output_fpath=./data/train.jsonl`.

#### Validation Error: Conflicting Options

```text
ValueError: Cannot specify both artifact_fpath and split
```

**Fix**: Use `artifact_fpath` for raw files OR `split` for structured datasets—not both.

---

## Private Repositories

> **Warning**
>
> Avoid passing tokens on the command line—they appear in shell history.

**Recommended** — Use environment variable:

```bash
export HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxx
gym dataset download \
    --repo-id my-org/private-dataset \
    --output-dir ./data/
```

Get your token at [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens). Use a **read-only** token.

#### Alternative: Pass token directly

Not recommended for shared systems:

```bash
gym dataset download \
    --repo-id my-org/private-dataset \
    --output-dir ./data/ \
    +hf_token=hf_xxxxxxxxxxxxxxxxxxxxxxxxx
```

#### Automatic Downloads During Data Preparation

NeMo Gym can automatically download missing datasets during data preparation. Declare a `source:` block (`type: huggingface`) in your resources server config:

```yaml
datasets:
  - name: train
    type: train
    jsonl_fpath: resources_servers/code_gen/data/train.jsonl
    source:
      type: huggingface
      repo_id: nvidia/nemotron-RL-coding-competitive_coding
      artifact_fpath: opencodereasoning_filtered_25k_train.jsonl
    license: Apache 2.0
```

Run with download enabled:

```bash
gym dataset collate \
    --resources-server code_gen \
    --output-dir ./data/prepared \
    --mode train_preparation \
    --download \
    +data_source=huggingface
```

If `jsonl_fpath` doesn't exist locally, NeMo Gym downloads from the dataset's `source:` (here, the Hugging Face `repo_id`) before processing. The legacy `huggingface_identifier:` block still works but is deprecated in favor of `source:`.

#### Caching Behavior

Downloads use Hugging Face's cache at `~/.cache/huggingface/`.

* **Structured datasets**: Reads from cache (fast), overwrites output file
* **Raw files**: Uses cached copy, then copies to output path

To force fresh download:

```bash
rm -rf ~/.cache/huggingface/hub/datasets--<org>--<dataset>
```

#### Source References

| Section          | Source                                 |
| ---------------- | -------------------------------------- |
| Config schema    | `nemo_gym/config_types.py:306-349`     |
| Download logic   | `nemo_gym/hf_utils.py:57-115`          |
| Validation rules | `nemo_gym/config_types.py:334-349`     |
| Auto-download    | `nemo_gym/train_data_utils.py:476-494` |

## Next Steps

#### [Prepare and Validate](/data/prepare-validate)

Preprocess raw data, run `gym dataset collate`, and add `agent_ref` routing.

#### [Training Tutorials](/tutorials/training-tutorials)

Use validated data to train with your preferred RL framework.