NeMo Usage

To use SentencePiece tokenizer in NeMo, set up the model configuration as follows:

tokenizer:
  library: sentencepiece
  type: null
  model: /Path/To/Tokenizer/tokenizer.model
  delimiter: null
  vocab_file: null
  merge_file: null
  sentencepiece_legacy: false