Important

NeMo 2.0 is an experimental feature and currently released in the dev container only: nvcr.io/nvidia/nemo:dev. Please refer to NeMo 2.0 overview for information on getting started.

NeMo Usage

To use SentencePiece tokenizer in NeMo, set up the model configuration as follows:

tokenizer:
  library: sentencepiece
  type: null
  model: /Path/To/Tokenizer/tokenizer.model
  delimiter: null
  vocab_file: null
  merge_file: null
  sentencepiece_legacy: false