Important

You are viewing the NeMo 2.0 documentation. This release introduces significant changes to the API and a new library, NeMo Run. We are currently porting all features from NeMo 1.0 to 2.0. For documentation on previous versions or features not yet available in 2.0, please refer to the NeMo 24.07 documentation.

Model NLP#

The config file for NLP models contain three main sections:

  • trainer: contains the configs for PTL training. For more information, refer to NeMo Models and PTL Trainer class API <https://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html#trainer-class-api>.

  • exp_manager: the configs of the experiment manager. For more information, refer to NeMo Models.

  • model: contains the configs of the datasets, model architecture, tokenizer, optimizer, scheduler, etc.

The following sub-sections of the model section are shared among most of the NLP models.

  • tokenizer: specifies the tokenizer

  • language_model: specifies the underlying model to be used as the encoder

  • optim: the configs of the optimizer and scheduler NeMo Models

The tokenizer and language_model sections have the following parameters:

Parameter

Data Type

Description

model.tokenizer.tokenizer_name

string

Tokenizer name will be filled automatically based on model.language_model.pretrained_model_name.

model.tokenizer.vocab_file

string

Path to tokenizer vocabulary.

model.tokenizer.tokenizer_model

string

Path to tokenizer model (only for sentencepiece tokenizer).

model.language_model.pretrained_model_name

string

Pre-trained language model name, for example: bert-base-cased or bert-base-uncased.

model.language_model.lm_checkpoint

string

Path to the pre-trained language model checkpoint.

model.language_model.config_file

string

Path to the pre-trained language model config file.

model.language_model.config

dictionary

Config of the pre-trained language model.

The parameter model.language_model.pretrained_model_name can be one of the following:

  • megatron-bert-345m-uncased, megatron-bert-345m-cased, biomegatron-bert-345m-uncased, biomegatron-bert-345m-cased, bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased

  • distilbert-base-uncased, distilbert-base-cased

  • roberta-base, roberta-large, distilroberta-base

  • albert-base-v1, albert-large-v1, albert-xlarge-v1, albert-xxlarge-v1, albert-base-v2, albert-large-v2, albert-xlarge-v2, albert-xxlarge-v2