core.datasets.retro.config.tokenizers#

Container class for GPT and Bert tokenizers.

Module Contents#

Classes#

RetroTokenizers

Container class for GPT and Bert tokenizers.

API#

class core.datasets.retro.config.tokenizers.RetroTokenizers#

Container class for GPT and Bert tokenizers.

gpt: megatron.core.tokenizers.MegatronTokenizerBase#

None

bert: megatron.core.tokenizers.MegatronTokenizerBase#

None