`bridge.data.energon.energon_provider`#

Module Contents#

Classes#

EnergonProvider

Energon Provider.

API#

class bridge.data.energon.energon_provider.EnergonProvider#

Bases: megatron.bridge.data.utils.DatasetProvider

Energon Provider.

path: str#: None

image_processor: Optional[Any]#: None

seq_length: int#: None

micro_batch_size: int#: None

global_batch_size: int#: None

num_workers: torch.int_repr#: None

dataloader_type: str#: ‘external’

task_encoder: Optional[Any]#: None

pack_sequences_in_batch: bool#: False

build_datasets( context: megatron.bridge.data.utils.DatasetBuildContext, )#