Is this page helpful?

`core.transformer.moe.moe_layer_config`#

Module Contents#

Classes#

MoELayerConfig

Configuration for a Mixture-of-Experts layer in a hybrid stack.

API#

class core.transformer.moe.moe_layer_config.MoELayerConfig#

Bases: megatron.core.transformer.transformer_config.TransformerConfig

Configuration for a Mixture-of-Experts layer in a hybrid stack.

Due to backwards-compatibility, this config’s arguments are defined in TransformerConfig.