> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.datasets.diffusion.meta_files_dataset

## Module Contents

### Classes

| Name                                                                                                    | Description                               |
| ------------------------------------------------------------------------------------------------------- | ----------------------------------------- |
| [`MetaFilesDataset`](#nemo_automodel-components-datasets-diffusion-meta_files_dataset-MetaFilesDataset) | PyTorch dataset for WAN2.1 `.meta` files. |

### Functions

| Name                                                                                                                          | Description                                                        |
| ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ |
| [`build_dataloader`](#nemo_automodel-components-datasets-diffusion-meta_files_dataset-build_dataloader)                       | Build a dataloader for pre-encoded diffusion metadata files.       |
| [`build_node_parallel_sampler`](#nemo_automodel-components-datasets-diffusion-meta_files_dataset-build_node_parallel_sampler) | Build a distributed sampler when torch.distributed is initialized. |
| [`collate_fn`](#nemo_automodel-components-datasets-diffusion-meta_files_dataset-collate_fn)                                   | Collate encoded video metadata samples into a training batch.      |
| [`create_dataloader`](#nemo_automodel-components-datasets-diffusion-meta_files_dataset-create_dataloader)                     | Create a default metadata dataloader for node-parallel loading.    |

### Data

[`logger`](#nemo_automodel-components-datasets-diffusion-meta_files_dataset-logger)

### API

```python
class nemo_automodel.components.datasets.diffusion.meta_files_dataset.MetaFilesDataset(
    meta_folder: str,
    transform_text: typing.Optional[typing.Callable[[torch.Tensor], torch.Tensor]] = None,
    transform_video: typing.Optional[typing.Callable[[torch.Tensor], torch.Tensor]] = None,
    filter_fn: typing.Optional[typing.Callable[[Dict], bool]] = None,
    device: str = 'cpu',
    max_files: typing.Optional[int] = None
)
```

**Bases:** `Dataset`

PyTorch dataset for WAN2.1 `.meta` files.

```python
nemo_automodel.components.datasets.diffusion.meta_files_dataset.MetaFilesDataset.__getitem__(
    index: int
) -> typing.Dict[str, torch.Tensor]
```

```python
nemo_automodel.components.datasets.diffusion.meta_files_dataset.MetaFilesDataset.__len__() -> int
```

```python
nemo_automodel.components.datasets.diffusion.meta_files_dataset.MetaFilesDataset._log_dataset_stats() -> None
```

```python
nemo_automodel.components.datasets.diffusion.meta_files_dataset.build_dataloader(
    meta_folder: str,
    batch_size: int,
    dp_rank: int,
    dp_world_size: int,
    shuffle: bool = True,
    num_workers: int = 2,
    device: str = 'cpu',
    transform_text: typing.Optional[typing.Callable[[torch.Tensor], torch.Tensor]] = None,
    transform_video: typing.Optional[typing.Callable[[torch.Tensor], torch.Tensor]] = None,
    filter_fn: typing.Optional[typing.Callable[[Dict], bool]] = None,
    max_files: typing.Optional[int] = None
) -> typing.Tuple[torch.utils.data.DataLoader, typing.Optional[torch.utils.data.DistributedSampler]]
```

Build a dataloader for pre-encoded diffusion metadata files.

```python
nemo_automodel.components.datasets.diffusion.meta_files_dataset.build_node_parallel_sampler(
    dataset: 'Dataset',
    dp_rank: int,
    dp_world_size: int,
    shuffle: bool = True
) -> typing.Optional['DistributedSampler']
```

Build a distributed sampler when torch.distributed is initialized.

```python
nemo_automodel.components.datasets.diffusion.meta_files_dataset.collate_fn(
    batch: typing.List[typing.Dict[str, torch.Tensor]]
) -> typing.Dict[str, torch.Tensor]
```

Collate encoded video metadata samples into a training batch.

```python
nemo_automodel.components.datasets.diffusion.meta_files_dataset.create_dataloader(
    meta_folder: str,
    batch_size: int,
    num_nodes: int
) -> typing.Tuple[torch.utils.data.DataLoader, typing.Optional[torch.utils.data.DistributedSampler]]
```

Create a default metadata dataloader for node-parallel loading.

```python
nemo_automodel.components.datasets.diffusion.meta_files_dataset.logger = logging.getLogger(__name__)
```