NeMo Text Processing Deployment

This tool deploys NeMo Inverse Text Normalization (ITN) and NeMo Text Normalization (TN) for production [TOOLS-ITN_DEPLOY3]. It uses Sparrowhawk [TOOLS-ITN_DEPLOY2] – an open-source version of Google Kestrel [TOOLS-ITN_DEPLOY1].

Requirements

  1. nemo_text_processing package

Usage

Starts docker container with production backend with plugged in grammars. This is entry point script. Set GRAMMARS=tn_grammars or GRAMMARS=itn_grammars to export either TN or ITN grammars from nemo_text_processing.

bash export_grammar.sh <GRAMMARS> [INPUT_CASE]

This script runs the following steps in sequence:

Exports grammar ClassifyFst and VerbalizeFst from nemo_text_processing to OUTPUT_DIR/classify/tokenize_and_classify.far and OUTPUT_DIR/verbalize/verbalize.far respectively.

python pynini_export.py <--output_dir OUTPUT_DIR> <--grammars GRAMMARS> <--input_case INPUT_CASE>

Builds C++ production backend docker

bash docker/build.sh

Plugs in grammars into production backend by mounting grammar directory classify/ and verbalize/ with sparrowhawk grammar directory inside docker. Returns docker prompt

bash docker/launch.sh

Runs TN or ITN in docker container:

echo "two dollars fifty" | ../../src/bin/normalizer_main --config=sparrowhawk_configuration.ascii_proto

This returns $2.50 for ITN.

References

TOOLS-ITN_DEPLOY1

Peter Ebden and Richard Sproat. The kestrel tts text normalization system. Natural Language Engineering, 21(3):333, 2015.

TOOLS-ITN_DEPLOY2

Alexander Gutkin, Linne Ha, Martin Jansche, Knot Pipatsrisawat, and Richard Sproat. Tts for low resource languages: a bangla synthesizer. In 10th Language Resources and Evaluation Conference. 2016.

TOOLS-ITN_DEPLOY3

Yang Zhang, Evelina Bakhturina, Kyle Gorman, and Boris Ginsburg. Nemo inverse text normalization: from development to production. 2021. arXiv:2104.05055.