Installation

Install from a C++ Source

To build and install the SentencePiece command line tools from C++ source on Ubuntu, use the following code:

% sudo apt-get install cmake build-essential pkg-config libgoogle-perftools-dev

% git clone https://github.com/google/sentencepiece.git
% cd sentencepiece
% mkdir build
% cd build
% cmake ..
% make -j $(nproc)
% sudo make install
% sudo ldconfig -v

Install from a Python Module

pip install sentencepiece

Please ensure that the SentencePiece Python module is pre-installed in the NeMo Framework container.

For more information on installation, please see this page.