core.models.bert.bert_layer_specs

Module Contents

Functions

get_bert_layer_with_transformer_engine_submodules	Use these submodules to use lower-level Transformer Engine modules (required for fp8 training).
get_bert_layer_with_transformer_engine_spec	Use this spec to use lower-level Transformer Engine modules (required for fp8 training).
__getattr__

Data

bert_layer_local_spec

API

core.models.bert.bert_layer_specs.get_bert_layer_with_transformer_engine_submodules() → megatron.core.transformer.transformer_layer.TransformerLayerSubmodules

Use these submodules to use lower-level Transformer Engine modules (required for fp8 training).

Returns:

Submodules with TE modules.

Return type:

TransformerLayerSubmodules

core.models.bert.bert_layer_specs.get_bert_layer_with_transformer_engine_spec()

Use this spec to use lower-level Transformer Engine modules (required for fp8 training).

Returns:

Module specification with TE modules

Return type:

ModuleSpec

core.models.bert.bert_layer_specs.__getattr__(name)

core.models.bert.bert_layer_specs.bert_layer_local_spec

‘ModuleSpec(…)’