Neural Text Normalization Models

Text normalization is the task of converting a written text into its spoken form. For example, $123 should be verbalized as one hundred twenty three dollars, while 123 King Ave should be verbalized as one twenty three King Avenue. At the same time, the inverse problem is about converting a spoken sequence (e.g., an ASR output) into its written form.

NeMo has an implementation that allows you to build a neural-based system that is able to do both text normalization (TN) and also inverse text normalization (ITN). At a high level, the system consists of two individual components:

• DuplexTaggerModel - a Transformer-based tagger for identifying “semiotic” spans in the input (e.g., spans that are about times, dates, or monetary amounts).

• DuplexDecoderModel - a Transformer-based seq2seq model for decoding the semiotic spans into their appropriate forms (e.g., spoken forms for TN and written forms for ITN).

The typical workflow is to first train a DuplexTaggerModel and also a DuplexDecoderModel. An example training script is provided: duplex_text_normalization_train.py. After that, the two trained models can be used to initialize a DuplexTextNormalizationModel that can be used for end-to-end inference. An example script for evaluation is provided here: duplex_text_normalization_test.py. The script for inference of the full pipeline is provided here: duplex_text_normalization_infer.py. This script runs inference from a raw text file or an interactive terminal. The term duplex refers to the fact that our system can be trained to do both TN and ITN. However, you can also specifically train the system for only one of the tasks.

To run the pretrained models interactively see Model Inference.

Available models

Both the DuplexTaggerModel model and the DuplexDecoderModel model use the same text format as the Google text normalization dataset. The data needs to be stored in TAB separated files (.tsv) with three columns. The first of which is the “semiotic class” (e.g., numbers, times, dates) , the second is the token in written form, and the third is the spoken form. An example sentence in the dataset is shown below. In the example, self denotes that the spoken form is the same as the written form.

            

            
PLAIN       The     <self>
PLAIN       company 's      <self>
PLAIN       revenues        <self>
PLAIN       grew    <self>
PLAIN       four    <self>
PLAIN       fold    <self>
PLAIN       between <self>
DATE        2005    two thousand five
PLAIN       and     <self>
DATE        2008    two thousand eight
PUNCT       .       <self>
<eos>       <eos>
        

More information about the Google text normalization dataset can be found in the paper RNN Approaches to Text Normalization: A Challenge [NLP-TEXTNORM2]. The script for splitting the Google text normalization data files into train, dev, test can be found here: data/data_split.py.

Processing scripts can be found in the same folder. Right now we only provide scripts for English text normalization, see data/en/data_preprocessing.py. The details can be found at the top of the scripts. The purpose of the preprocessing scripts is to standardize the format in order to help with model training. We also changed punctuation class PUNCT to be treated like a plain token ( label changed from <sil> to ``<self>), since we want to preserve punctuation even after normalization. For text normalization it is crucial to avoid unrecoverable errors, which are linguistically coherent and not semantic preserving. We noticed that due to data scarcity the model struggles verbalizing long numbers correctly, so we changed the ground truth for long numbers to digit by digit verbalization. We also ignore certain semiotic classes from neural verbalization, e.g. ELECTRONIC or WHITELIST – VERBATIM and LETTER in the original dataset. Instead we label urls/email addresses and abbreviations as plain tokens, and handle it separately with WFST-based grammars, see Model Inference. This simplifies the task for the model and significantly reduces unrecoverable errors.

Data upsampling is an effective way to increase the training data for better model performance, especially on the long tail of semiotic tokens. We used upsampling for training an English text normalization model, see data/en/upsampling.py. Currently this script only upsamples a few classes, that are diverse in semiotic tokens but at the same time underrepresented in the training data. Of all the input files in train folder created by data/data_split.py. this script takes the first file and detects the class patterns that occur in it. For those that are underrepresented, quantitatively defined as lower than min_number, the other files are scanned for sentences that have the missing patterns. Those sentences are appended to the first file, which can then be used for training. Details can be found at the top of the script.

When training with DistributedDataParallel, each process has its own copy of the dataset. For large datasets, this may not always fit in CPU memory. Webdatasets circumvents this problem by efficiently iterating over tar files stored on disk. Each tar file can contain hundreds to thousands of pickle files, each containing a single minibatch.

Tarred datasets can be created as follows:

            

            
python examples/nlp/duplex_text_normalization/data/create_tarred_dataset.py \
    --input_files = "<trained_processed/output-00099-of-00100>" \
    --input_files = "<trained_processed/output-00098-of-00100>" \
    --batch_size = "<batch size>" \
    --out_dir= "<TARRED_DATA_OUTPUT_DIR>"
        

An example training script is provided: duplex_text_normalization_train.py. The config file used for the example is at duplex_tn_config.yaml. You can change any of these parameters directly from the config file or update them with the command-line arguments.

The config file contains three main sections. The first section contains the configs for the tagger, the second section is about the decoder, and the last section is about the dataset. Most arguments in the example config file are quite self-explanatory (e.g., decoder_model.optim.lr refers to the learning rate for training the decoder). We have set most of the hyper-parameters to be the values that we found to be effective (for the English and the Russian subsets of the Google TN dataset). Some arguments that you may want to modify are:

• lang: The language of the dataset.

• mode: tn, itn or joint for text normalization, inverse text normalization or duplex mode

• data.train_ds.data_path: The path to the training file.

• data.validation_ds.data_path: The path to the validation file.

• data.test_ds.data_path: The path to the test file.

• data.test_ds.data_path: The path to the test file.

• data.test_ds.errors_log_fp: Path to the file for logging the errors for the test file.

• tagger_pretrained_model: pretrained model path or name (optional)

• decoder_pretrained_model: pretrained model path or name (optional)

• tagger_model.nemo_path: This is the path where the final trained tagger model will be saved to.

• decoder_model.nemo_path: This is the path where the final trained decoder model will be saved to.

• tagger_model.transformer: The huggingface transformer model used to initialize the tagger model weights

• decoder_model.transformer: The huggingface transformer model used to initialize the decoder model weights

Example of a training command:

            

            
python examples/nlp/duplex_text_normalization/duplex_text_normalization_train.py \
    data.base_dir=<PATH_TO_DATASET_DIR> \
    mode={tn,itn,joint}
        

There are 3 different modes. “tn” mode is for training a system for TN only. “itn” mode is for training a system for ITN. “joint” is for training a system that can do both TN and ITN at the same time. Note that the above command will first train a tagger and then train a decoder sequentially.

You can also train only a tagger (without training a decoder) by running the following command:

            

            
python examples/nlp/duplex_text_normalization/duplex_text_normalization_train.py \
    data.base_dir=PATH_TO_DATASET_DIR \
    mode={tn,itn,joint} \
    decoder_model.do_training=false
        

Or you can also train only a decoder (without training a tagger):

            

            
python examples/nlp/duplex_text_normalization/duplex_text_normalization_train.py \
    data.base_dir=PATH_TO_DATASET_DIR \
    mode={tn,itn,joint} \
    tagger_model.do_training=false
        

To use the tarred version of the data with the decoder model, set data.train_ds.use_tarred_dataset to True and provide path to the metadata.json file. The metadata file is created during the tarred dataset construction and stored at <TARRED_DATA_OUTPUT_DIR>. To enable training with the tarred dataset, add the following arguments:

            

            
data.train_ds.use_tarred_dataset=True \
data.train_ds.tar_metadata_file=\PATH_TO\<TARRED_DATA_OUTPUT_DIR>\metadata.json
        

Run the full inference pipeline:

            

            
cd NeMo/examples/nlp/duplex_text_normalization;

# run inference in interactive mode using pretrained tagger and decoder models
python duplex_text_normalization_infer.py \
    tagger_pretrained_model=neural_text_normalization_t5 \
    decoder_pretrained_model=neural_text_normalization_t5 \
    inference.from_file=False \
    lang=en \
    mode=tn
        

To run inference from a file adjust the previous command by

            

            
inference.from_file=<path_to_file>
inference.interactive=False
        

This pipeline consists of

The tagger model first uses a Transformer encoder (e.g., albert-base-v2) to build a contextualized representation for each input token. It then uses a classification head to predict the tag for each token (e.g., if a token should stay the same, its tag should be SAME). The decoder model then takes the semiotic spans identified by the tagger and transform them into the appropriate forms (e.g., spoken forms for TN and written forms for ITN). The decoder model is essentially a Transformer-based encoder-decoder seq2seq model (e.g., the example training script uses the T5-base model by default). Overall, our design is partly inspired by the RNN-based sliding window model proposed in the paper Neural Models of Text Normalization for Speech Applications [NLP-TEXTNORM4].

We introduce a simple but effective technique to allow our model to be duplex. Depending on the task the model is handling, we append the appropriate prefix to the input. For example, suppose we want to transform the text I live in 123 King Ave to its spoken form (i.e., TN problem), then we will simply append the prefix tn to it and so the final input to our models will actually be tn I live in tn 123 King Ave. Similarly, for the ITN problem, we just append the prefix itn to the input.

To improve the effectiveness and robustness of our models, we also experiment with some simple data augmentation techniques during training.

Data Augmentation for Training DuplexTaggerModel (Set to be False by default)

In the Google English TN training data, about 93% of the tokens are not in any semiotic span. In other words, the ground-truth tags of most tokens are of trivial types (i.e., SAME and PUNCT). To alleviate this class imbalance problem, for each original instance with several semiotic spans, we create a new instance by simply concatenating all the semiotic spans together. For example, considering the following ITN instance:

Original instance: [The|SAME] [revenues|SAME] [grew|SAME] [a|SAME] [lot|SAME] [between|SAME] [two|B-TRANSFORM] [thousand|I-TRANSFORM] [two|I-TRANSFORM] [and|SAME] [two|B-TRANSFORM] [thousand|I-TRANSFORM] [five|I-TRANSFORM] [.|PUNCT]

Augmented instance: [two|B-TRANSFORM] [thousand|I-TRANSFORM] [two|I-TRANSFORM] [two|B-TRANSFORM] [thousand|I-TRANSFORM] [five|I-TRANSFORM]

The argument data.train_ds.tagger_data_augmentation in the config file controls whether this data augmentation will be enabled or not.

Data Augmentation for Training DuplexDecoderModel (Set to be True by default)

Since the tagger may not be perfect, the inputs to the decoder may not all be semiotic spans. Therefore, to make the decoder become more robust against the tagger’s potential errors, we train the decoder with not only semiotic spans but also with some other more “noisy” spans. This way even if the tagger makes some errors, there will still be some chance that the final output is still correct.

The argument data.train_ds.decoder_data_augmentation in the config file controls whether this data augmentation will be enabled or not.

[NLP-TEXTNORM1]

Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. Moses: open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, 177–180. Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL: https://aclanthology.org/P07-2045.

[NLP-TEXTNORM2]

Eric Malmi, Sebastian Krause, Sascha Rothe, Daniil Mirylenka, and Aliaksei Severyn. Encode, tag, realize: high-precision text editing. arXiv preprint arXiv:1909.01187, 2019.

[NLP-TEXTNORM3]

Richard Sproat and Navdeep Jaitly. Rnn approaches to text normalization: a challenge. arXiv preprint arXiv:1611.00068, 2016.

[NLP-TEXTNORM4]

Hao Zhang, R. Sproat, Axel H. Ng, Felix Stahlberg, Xiaochang Peng, Kyle Gorman, and B. Roark. Neural models of text normalization for speech applications. Computational Linguistics, pages 293–338, 2019.