`core.datasets.retro.config.tokenizers`#

Container class for GPT and Bert tokenizers.

Module Contents#

Classes#

RetroTokenizers

Container class for GPT and Bert tokenizers.

API#

class core.datasets.retro.config.tokenizers.RetroTokenizers#

Container class for GPT and Bert tokenizers.

gpt: megatron.core.tokenizers.MegatronTokenizerBase#: None

bert: megatron.core.tokenizers.MegatronTokenizerBase#: None