> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.train_data_utils

## Module Contents

### Classes

| Name                                                                              | Description                                                                         |
| --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| [`Accumulator`](#nemo_gym-train_data_utils-Accumulator)                           | -                                                                                   |
| [`AvgMinMax`](#nemo_gym-train_data_utils-AvgMinMax)                               | -                                                                                   |
| [`CategoricalMetrics`](#nemo_gym-train_data_utils-CategoricalMetrics)             | Accumulate a bounded categorical distribution.                                      |
| [`DatasetMetrics`](#nemo_gym-train_data_utils-DatasetMetrics)                     | -                                                                                   |
| [`DatasetValidatorState`](#nemo_gym-train_data_utils-DatasetValidatorState)       | -                                                                                   |
| [`StringMetrics`](#nemo_gym-train_data_utils-StringMetrics)                       | -                                                                                   |
| [`TrainDataProcessor`](#nemo_gym-train_data_utils-TrainDataProcessor)             | -                                                                                   |
| [`TrainDataProcessorConfig`](#nemo_gym-train_data_utils-TrainDataProcessorConfig) | Prepare and validate training data, generating metrics and statistics for datasets. |

### Functions

| Name                                                                                      | Description                                                                                  |
| ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| [`_materialized_task_parts`](#nemo_gym-train_data_utils-_materialized_task_parts)         | -                                                                                            |
| [`_observe_task_metric`](#nemo_gym-train_data_utils-_observe_task_metric)                 | -                                                                                            |
| [`aggregate_other_metrics`](#nemo_gym-train_data_utils-aggregate_other_metrics)           | Combines misc items (those other than response/response create params) into current metrics  |
| [`compute_sample_metrics`](#nemo_gym-train_data_utils-compute_sample_metrics)             | -                                                                                            |
| [`postprocess_other_metrics`](#nemo_gym-train_data_utils-postprocess_other_metrics)       | Aggregates metrics and merges current metrics (containing only AvgMinMax) with StringMetrics |
| [`validate_backend_credentials`](#nemo_gym-train_data_utils-validate_backend_credentials) | Check if required env variables are present for the chosen backend                           |

### Data

[`MAX_CATEGORICAL_LABELS`](#nemo_gym-train_data_utils-MAX_CATEGORICAL_LABELS)

[`__getattr__`](#nemo_gym-train_data_utils-__getattr__)

### API

```python
class nemo_gym.train_data_utils.Accumulator()
```

**Bases:** `BaseModel`

**`is_aggregated`** `bool = Field(default=False, exclude=True)`

---

```python
nemo_gym.train_data_utils.Accumulator._add(
    other: typing.Self
) -> None
```

abstract

```python
nemo_gym.train_data_utils.Accumulator._aggregate() -> typing.Self
```

abstract

```python
nemo_gym.train_data_utils.Accumulator.add(
    other: typing.Self
) -> None
```

```python
nemo_gym.train_data_utils.Accumulator.aggregate() -> typing.Self
```

```python
class nemo_gym.train_data_utils.AvgMinMax()
```

**Bases:** [Accumulator](#nemo_gym-train_data_utils-Accumulator)

**`M2`** `float = Field(default=0, exclude=True)`

---

**`average`** `float = Field(serialization_alias='Average', default=0)`

---

**`max`** `float`

---

**`mean`** `float = Field(default=0, exclude=True)`

---

**`min`** `float`

---

**`model_config`** `= ConfigDict(arbitrary_types_allowed=True)`

---

**`stddev`** `float`

---

**`total`** `int`

---

```python
nemo_gym.train_data_utils.AvgMinMax._add(
    other: typing.Self
) -> None
```

```python
nemo_gym.train_data_utils.AvgMinMax._aggregate() -> typing.Self
```

```python
nemo_gym.train_data_utils.AvgMinMax.observe(
    x: float
) -> None
```

```python
class nemo_gym.train_data_utils.CategoricalMetrics()
```

**Bases:** [Accumulator](#nemo_gym-train_data_utils-Accumulator)

Accumulate a bounded categorical distribution.

**`counts`** `Dict[str, int] = Field(default_factory=dict)`

---

**`other_count`** `int = Field(default=0, exclude=True)`

---

**`total_count`** `int = 0`

---

**`unique_count`** `int = 0`

---

```python
nemo_gym.train_data_utils.CategoricalMetrics._add(
    other: typing.Self
) -> None
```

```python
nemo_gym.train_data_utils.CategoricalMetrics._aggregate() -> typing.Self
```

```python
nemo_gym.train_data_utils.CategoricalMetrics.observe(
    value: str
) -> None
```

```python
class nemo_gym.train_data_utils.DatasetMetrics()
```

**Bases:** [Accumulator](#nemo_gym-train_data_utils-Accumulator)

**`json_dumped_number_of_words`** `AvgMinMax`

---

**`model_config`** `= ConfigDict(extra='allow')`

---

**`number_of_examples`** `int`

---

**`number_of_tasks`** `int`

---

**`number_of_tools`** `AvgMinMax`

---

**`number_of_turns`** `AvgMinMax`

---

**`task_input_json_dumped_number_of_words`** `AvgMinMax`

---

**`temperature`** `AvgMinMax`

---

```python
nemo_gym.train_data_utils.DatasetMetrics._add(
    other: typing.Self
) -> None
```

```python
nemo_gym.train_data_utils.DatasetMetrics._aggregate() -> typing.Self
```

```python
nemo_gym.train_data_utils.DatasetMetrics.model_dump_for_output() -> typing.Dict[str, typing.Any]
```

Serialize existing metrics, adding task metrics only for materialized rows.

```python
class nemo_gym.train_data_utils.DatasetValidatorState()
```

**Bases:** `BaseModel`

**`key_counts`** `Counter = Field(default_factory=Counter)`

---

**`metrics`** `DatasetMetrics = Field(default_factory=DatasetMetrics)`

---

**`model_config`** `= ConfigDict(arbitrary_types_allowed=True)`

---

**`offending_example_idxs`** `List[int] = Field(default_factory=list)`

---

**`other_metrics`** `Dict[str, Any] = Field(default_factory=dict)`

---

```python
class nemo_gym.train_data_utils.StringMetrics()
```

**Bases:** `BaseModel`

**`total_count`** `int`

---

**`unique_count`** `int`

---

```python
class nemo_gym.train_data_utils.TrainDataProcessor()
```

**Bases:** `BaseModel`

```python
nemo_gym.train_data_utils.TrainDataProcessor._collate_samples_single_type(
    type: nemo_gym.config_types.DatasetType,
    server_instance_configs: typing.List[nemo_gym.config_types.ServerInstanceConfig],
    task_data_validation: str = 'warn'
) -> typing.List[pathlib.Path]
```

```python
nemo_gym.train_data_utils.TrainDataProcessor._dataset_metrics_hook_for(
    c: nemo_gym.config_types.ServerInstanceConfig,
    server_instance_configs: typing.List[nemo_gym.config_types.ServerInstanceConfig]
) -> nemo_gym.dataset_metrics.DatasetMetricHook | None
```

classmethod

```python
nemo_gym.train_data_utils.TrainDataProcessor._iter_dataset_lines(
    dataset_config: nemo_gym.config_types.DatasetConfig
)
```

```python
nemo_gym.train_data_utils.TrainDataProcessor._owning_resources_server_impl(
    c: nemo_gym.config_types.ServerInstanceConfig,
    server_instance_configs: typing.List[nemo_gym.config_types.ServerInstanceConfig]
) -> typing.Optional[str]
```

staticmethod

The resources-server implementation (directory) that owns a declaring instance's data.

Datasets declared by a resources server belong to its own implementation; datasets
declared by an agent belong to the resources server the agent references. Self-contained
agents (no resources\_server reference) return None here; the validator lookup then falls
back to the agent's own directory.

```python
nemo_gym.train_data_utils.TrainDataProcessor._print_title(
    title: str
) -> None
```

```python
nemo_gym.train_data_utils.TrainDataProcessor._task_data_validator_for(
    c: nemo_gym.config_types.ServerInstanceConfig,
    d: typing.Union[nemo_gym.config_types.DatasetConfig, nemo_gym.config_types.BenchmarkDatasetConfig],
    server_instance_configs: typing.List[nemo_gym.config_types.ServerInstanceConfig]
) -> typing.Optional[nemo_gym.task_data.TaskDataValidator]
```

classmethod

```python
nemo_gym.train_data_utils.TrainDataProcessor._validate_aggregate_metrics(
    aggregate_metrics_dict: typing.Dict,
    metrics_fpath: pathlib.Path
) -> typing.Optional[pathlib.Path]
```

Returns the conflicting metrics fpath if invalid. Else returns None

```python
nemo_gym.train_data_utils.TrainDataProcessor._validate_samples_and_aggregate_metrics_single_dataset(
    dataset_config: nemo_gym.config_types.DatasetConfig,
    dataset_metrics_hook: nemo_gym.dataset_metrics.DatasetMetricHook | None = None
) -> nemo_gym.train_data_utils.DatasetValidatorState
```

```python
nemo_gym.train_data_utils.TrainDataProcessor._validate_samples_and_aggregate_metrics_single_sample(
    state: nemo_gym.train_data_utils.DatasetValidatorState,
    sample_idx: int,
    sample_dict_str: str,
    require_responses: bool = True,
    dataset_metrics_hook: nemo_gym.dataset_metrics.DatasetMetricHook | None = None
) -> None
```

```python
nemo_gym.train_data_utils.TrainDataProcessor.collate_samples(
    config: nemo_gym.train_data_utils.TrainDataProcessorConfig,
    server_instance_configs: typing.List[nemo_gym.config_types.ServerInstanceConfig],
    dataset_type_to_aggregate_metrics: typing.Dict[str, nemo_gym.train_data_utils.DatasetMetrics]
) -> None
```

```python
nemo_gym.train_data_utils.TrainDataProcessor.load_and_validate_server_instance_configs(
    config: nemo_gym.train_data_utils.TrainDataProcessorConfig,
    global_config_dict: omegaconf.DictConfig
) -> typing.List[nemo_gym.config_types.ServerInstanceConfig]
```

```python
nemo_gym.train_data_utils.TrainDataProcessor.load_datasets(
    config: nemo_gym.train_data_utils.TrainDataProcessorConfig,
    server_instance_configs: typing.List[nemo_gym.config_types.ServerInstanceConfig]
) -> None
```

```python
nemo_gym.train_data_utils.TrainDataProcessor.run(
    global_config_dict: omegaconf.DictConfig
)
```

See the README section "How To: Prepare and validate data for PR submission or RL training"

```python
nemo_gym.train_data_utils.TrainDataProcessor.validate_samples_and_aggregate_metrics(
    server_instance_configs: typing.List[nemo_gym.config_types.ServerInstanceConfig],
    overwrite_metrics_conflicts: bool
) -> typing.Dict[str, nemo_gym.train_data_utils.DatasetMetrics]
```

```python
class nemo_gym.train_data_utils.TrainDataProcessorConfig()
```

**Bases:** [BaseNeMoGymCLIConfig](/nemo/gym/nemo-gym/nemo_gym/config_types#nemo_gym-config_types-BaseNeMoGymCLIConfig)

Prepare and validate training data, generating metrics and statistics for datasets.

Examples:

```python
config_paths="resources_servers/example_multi_step/configs/example_multi_step.yaml,\
responses_api_models/openai_model/configs/openai_model.yaml"
gym dataset collate "+config_paths=[${config_paths}]"         +output_dirpath=data/example_multi_step         +mode=example_validation
```

**`data_source`** `Literal['gitlab', 'huggingface']`

---

**`effective_task_data_validation`** `str`

---

**`in_scope_dataset_types`** `List[DatasetType]`

---

**`mode`** `Union[Literal['train_preparation'], Literal['example_validation']]`

---

**`output_dirpath`** `str`

---

**`overwrite_metrics_conflicts`** `bool`

---

**`should_download`** `bool`

---

**`task_data_validation`** `Literal['off', 'warn', 'error', 'auto']`

---

```python
nemo_gym.train_data_utils._materialized_task_parts(
    sample: collections.abc.Mapping[str, typing.Any]
) -> tuple[nemo_gym.episode_types.TaskId, collections.abc.Mapping[str, typing.Any]] | None
```

```python
nemo_gym.train_data_utils._observe_task_metric(
    metrics: nemo_gym.train_data_utils.DatasetMetrics,
    name: str,
    value: nemo_gym.dataset_metrics.DatasetMetricValue
) -> None
```

```python
nemo_gym.train_data_utils.aggregate_other_metrics(
    metrics: typing.Dict[str, typing.Any],
    sample: typing.Dict[str, typing.Any]
) -> None
```

Combines misc items (those other than response/response create params) into current metrics

```python
nemo_gym.train_data_utils.compute_sample_metrics(
    sample_dict_str: str,
    require_responses: bool = True
) -> typing.Tuple[nemo_gym.train_data_utils.DatasetMetrics, bool]
```

```python
nemo_gym.train_data_utils.postprocess_other_metrics(
    metrics: nemo_gym.train_data_utils.DatasetMetrics,
    other_metrics: typing.Dict[str, typing.Any]
) -> None
```

Aggregates metrics and merges current metrics (containing only AvgMinMax) with StringMetrics

```python
nemo_gym.train_data_utils.validate_backend_credentials(
    backend: str
) -> tuple[bool, str]
```

Check if required env variables are present for the chosen backend

```python
nemo_gym.train_data_utils.MAX_CATEGORICAL_LABELS = 100
```

```python
nemo_gym.train_data_utils.__getattr__ = moved_attr_getter(__name__, {'prepare_data': 'nemo_gym.cli.dataset'})
```