Parallelisms#
NeMo Megatron supports 4 types of parallelisms (can be mixed together arbitraritly):
Distributed Data parallelism#

Tensor Parallelism#

Pipeline Parallelism#

Sequence Parallelism#

Parallelism nomenclature#
When reading and modifying NeMo Megatron code you will encounter the following terms.
