NeMo Usage
To use SentencePiece tokenizer in NeMo, set up the model configuration as follows:
tokenizer:
library: sentencepiece
type: null
model: /Path/To/Tokenizer/tokenizer.model
delimiter: null
vocab_file: null
merge_file: null
sentencepiece_legacy: false