Parallelisms#

NeMo Megatron supports 4 types of parallelisms (can be mixed together arbitraritly):

Distributed Data parallelism#

Distributed Data Parallel

Tensor Parallelism#

Tensor Parallel

Pipeline Parallelism#

Pipeline Parallel

Sequence Parallelism#

Sqeuence Parallel

Parallelism nomenclature#

When reading and modifying NeMo Megatron code you will encounter the following terms.

Parallelism nomenclature