Megatron Core is a Python library that has the core components required to build your language models. A reference implementation of Megatron Core can be found in NeMo It offers a simple and intuitive API.
User Guide
API Guide
- API Guide
- models package
- tensor_parallel package
- context_parallel package
- pipeline_parallel package
- fusions package
- transformer package
- Submodules
- transformer.attention module
- transformer.dot_product_attention module
- transformer.enums module
- transformer.identity_op module
- transformer.mlp module
- transformer.module module
- transformer.transformer_block module
- transformer.transformer_config module
- transformer.transformer_layer module
- transformer.utils module
- Module contents
- Mixture of Experts package
- dist_checkpointing package
- Distributed Optimizer
- distributed package
- datasets package
- Data Pipeline
- Submodules
- datasets.blended_megatron_dataset_config module
- datasets.blended_megatron_dataset_builder module
- datasets.megatron_tokenizer module
- datasets.indexed_dataset module
- datasets.megatron_dataset module
- datasets.gpt_dataset module
- datasets.masked_dataset module
- datasets.bert_dataset module
- datasets.t5_dataset module
- datasets.blended_dataset module
- datasets.utils module
- Module contents
- Microbatches Calculator
- Optimizer Parameters Scheduler
- encoder-decoder-parallelism package
- models package