Known Issues

Fixes for the following issues will be released shortly:

  • CUDNN Flash attention disabled by default until further testing is done

  • For data preparation of GPT models, use your own dataset or an online dataset legally approved by your organization

  • Megatron Core gradient clipping issue when using MoE and EP

  • Race condition in NeMo experiment manager

  • Mistral & Mixtral tokenizers require Hugging Face login

  • Gemma, Starcoder, and Falcon 7B export to TRT-LLM only works with a single GPU and if the user wants to export, there is no descriptive error message shown to the user.

  • The following notebooks have functional issues and will be fixed in the next release

    • ASR_with_NeMo.ipynb

    • ASR_with_Subword_Tokenization.ipynb

    • AudioTranslationSample.ipynb

    • Megatron_Synthetic_Tabular_Data_Generation.ipynb

    • SpellMapper_English_ASR_Customization.ipynb

    • FastPitch_ChineseTTS_Training.ipynb

    • NeVA Tutorial.ipynb