Large Language Models

The NVIDIA NeMo™ Framework has everything needed to train Large Language Models. This includes setting up the compute cluster, downloading data, and selecting model hyperparameters. The default configurations for each model and task are tested on a regular basis and every configuration can be modified in order to train on new datasets or test new model hyperparameters.

You can find more details that are applicable to all models below.

• All Models
  
  • Prepare Environment
  • Data Preparation
  • Monitoring and Logging
  • AutoConfigurator
  • Resuming Training with a Different Number of Nodes
  • Generalized PEFT Framework
  • Exporting NeMo Models to TensorRT-LLM

Additionally, NVIDIA NeMo™ Framework has seamless integrations for several community large language models, providing access to a comprehensive suite of tools for everything from training to deployment. For model specific details, see the relevant section below.

• Llama and CodeLlama
• Gemma and CodeGemma
• Baichuan 2
• Falcon
• Mistral
• Mixtral
• StarCoder2
• T5
• mT5
• GPT
• BERT

For each model, you will find comprehensive instructions on:

• Data preparation

• Training & Evaluation

• Parameter Efficient Fine-Tuning (PEFT)

• Exporting the Model to TensorRT-LLM

• Deployment