Training an Acoustic Model with Subword Tokenization
Contents
Training an Acoustic Model with Subword Tokenization#
In this notebook, we train an ASR model for German, using the Citrinet model with cross language transfer learning. The workflow is demonstrated in the figure below.
We first demonstrate the training process with NeMo on 1 GPU in this notebook. To speed up training, multiple GPUs should be leveraged using the more efficient DDP (distributed data parallel) protocol, which must run in a seperate training script.
This notebook can be run from within the NeMo container, such as:
docker run --ipc=host --gpus=all --net=host --rm -it -v $PWD:/myworkspace nvcr.io/nvidia/nemo:22.08 bash
Note: PyTorch uses shared memory to share data between processes, so if torch multiprocessing is used (e.g. for multithreaded data loaders) the default shared memory segment size that container runs with is not enough, and you should increase shared memory size either with –ipc=host or –shm-size command line options to nvidia-docker run.
import nemo
import nemo.collections.asr as nemo_asr
print(nemo.__version__)
from hydra import initialize, initialize_config_module, initialize_config_dir, compose
from omegaconf import OmegaConf, open_dict
from pytorch_lightning.callbacks import ModelCheckpoint
import pytorch_lightning as pl
[NeMo W 2022-10-25 00:26:25 optimizers:77] Could not import distributed_fused_adam optimizer from Apex
OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
1.12.0
Cross-Language Transfer Learning#
Transfer learning is an important machine learning technique that uses a model’s knowledge of one task to perform better on another. Fine-tuning is one of the techniques to perform transfer learning. It is an essential part of the recipe for many state-of-the-art results where a base model is first pretrained on a task with abundant training data and then fine-tuned on different tasks of interest where the training data is less abundant or even scarce.
Transfer learning with NeMo is simple.
First, let’s load the pretrained Nemo Citrinet model, which was trained on ~6000 hours of English data.
import nemo.collections.asr as nemo_asr
asr_model = nemo_asr.models.EncDecCTCModelBPE.from_pretrained(model_name="stt_en_citrinet_1024")
[NeMo I 2022-10-25 00:26:35 cloud:56] Found existing object /root/.cache/torch/NeMo/NeMo_1.12.0/stt_en_citrinet_1024/86acfaf495a53383369fb6c9c547b8dd/stt_en_citrinet_1024.nemo.
[NeMo I 2022-10-25 00:26:35 cloud:62] Re-using file from: /root/.cache/torch/NeMo/NeMo_1.12.0/stt_en_citrinet_1024/86acfaf495a53383369fb6c9c547b8dd/stt_en_citrinet_1024.nemo
[NeMo I 2022-10-25 00:26:35 common:910] Instantiating model from pre-trained checkpoint
[NeMo I 2022-10-25 00:26:39 mixins:170] Tokenizer SentencePieceTokenizer initialized with 1024 tokens
[NeMo W 2022-10-25 00:26:39 modelPT:142] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
Train config :
manifest_filepath: null
sample_rate: 16000
batch_size: 32
trim_silence: true
max_duration: 16.7
shuffle: true
is_tarred: false
tarred_audio_filepaths: null
use_start_end_token: false
[NeMo W 2022-10-25 00:26:39 modelPT:149] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s).
Validation config :
manifest_filepath: null
sample_rate: 16000
batch_size: 32
shuffle: false
use_start_end_token: false
[NeMo W 2022-10-25 00:26:39 modelPT:155] Please call the ModelPT.setup_test_data() or ModelPT.setup_multiple_test_data() method and provide a valid configuration file to setup the test data loader(s).
Test config :
manifest_filepath: null
sample_rate: 16000
batch_size: 32
shuffle: false
use_start_end_token: false
[NeMo I 2022-10-25 00:26:40 features:225] PADDING: 16
[NeMo I 2022-10-25 00:26:44 save_restore_connector:243] Model EncDecCTCModelBPE was successfully restored from /root/.cache/torch/NeMo/NeMo_1.12.0/stt_en_citrinet_1024/86acfaf495a53383369fb6c9c547b8dd/stt_en_citrinet_1024.nemo.
Update vocabulary#
Next, check what kind of vocabulary/alphabet the model has right now
print(asr_model.decoder.vocabulary)
['<unk>', 's', '▁the', 't', '▁a', '▁i', "'", '▁and', '▁to', 'ed', 'd', '▁of', 'e', '▁in', 'ing', '.', '▁it', '▁you', 'n', '▁that', 'm', 'y', 'er', '▁he', 're', 'r', '▁was', '▁is', '▁for', '▁know', 'a', 'p', 'c', ',', '▁be', 'o', '▁but', '▁they', 'g', '▁so', 'ly', 'b', '▁s', '▁yeah', '▁we', '▁have', '▁re', '▁like', 'l', '▁on', 'll', 'u', '▁with', '▁do', 'al', '▁not', '▁are', 'or', 'ar', 'le', '▁this', '▁as', 'es', '▁c', '▁de', 'f', 'in', 'i', 've', '▁uh', 'ent', '▁or', '▁what', '▁me', '▁t', '▁at', '▁my', '▁his', '▁there', 'w', '▁all', '▁just', 'h', '▁can', 'ri', 'il', 'k', 'ic', '▁e', '▁', '▁um', '▁don', '▁b', '▁had', 'ch', 'ation', 'en', 'th', '▁no', '▁she', 'it', '▁one', '▁think', '▁st', '▁if', '▁from', 'ter', '▁an', 'an', 'ur', '▁out', 'on', '▁go', 'ck', '▁would', '▁were', '▁w', '▁will', '▁about', '▁right', 'ment', '▁her', 'te', 'ion', '▁well', '▁by', 'ce', '▁g', '▁oh', '▁up', 'ro', 'ra', '▁when', '▁some', '▁also', '▁their', 'ers', 'ow', '▁more', '▁time', 'ate', '▁has', '▁people', '▁see', '▁pa', 'el', '▁get', '▁ex', '▁mean', 'li', '▁really', 'v', '▁ra', '▁been', '▁said', '-', 'la', 'ge', '▁how', '▁po', 'ir', '▁mo', '▁who', '▁because', '▁co', '▁other', '▁f', 'id', 'ol', '▁un', '▁now', '▁work', 'ist', 'us', '▁your', '▁them', 'ver', 'as', 'ne', '▁ca', 'lo', '▁fa', '▁him', 'ng', '▁good', '▁could', '▁pro', 'ive', '▁con', 'de', 'un', 'age', '▁ma', '?', 'at', '▁ro', '▁ba', '▁then', '▁com', 'est', 'vi', '▁dis', 'ies', 'ance', '▁su', '▁even', '▁any', 'ut', 'ad', 'ul', '▁se', '▁two', '▁bu', '▁lo', '▁say', '▁la', '▁fi', 'is', '▁li', '▁over', '▁new', '▁man', '▁sp', 'ity', '▁did', '▁bo', '▁very', 'x', 'end', '▁which', '▁our', '▁after', '▁o', 'ke', '▁p', 'im', '▁want', '▁ha', '▁v', 'z', '▁where', 'ard', 'um', '▁into', 'ru', '▁di', '▁lot', '▁dr', 'mp', '▁day', 'ated', 'ci', '▁these', '▁than', '▁take', '▁kind', '▁got', 'ight', '▁make', 'ence', '▁pre', '▁going', 'ish', '▁k', 'able', '▁look', 'ti', 'per', '▁here', '▁en', '▁ah', 'ry', '▁too', '▁part', 'ant', 'one', '▁ho', '▁much', '▁way', '▁sa', '▁something', 'mo', '▁us', '▁th', '▁mhm', '▁mi', '▁off', 'pe', '▁back', 'les', '▁cr', '▁ri', '▁fe', 'und', '▁fl', 'port', '▁school', '▁ch', '▁should', '▁first', '▁only', '▁le', 'ot', 'tion', '▁little', '▁da', '▁hu', '▁d', 'me', 'ta', '▁down', '▁okay', '▁come', 'ain', 'ff', '▁car', 'co', '▁need', 'ture', '▁many', '▁things', '▁ta', 'qu', 'man', 'ty', 'iv', '▁year', 'he', '▁thing', 'ho', '▁singapore', 'po', '▁vi', '▁sc', '▁still', 'der', '▁hi', '▁never', '▁qu', 'ia', '▁fr', '▁min', '▁most', 'om', 'ful', '▁bi', '▁long', 'ig', '▁years', 'ous', '▁three', '▁play', '▁before', '▁pi', 'ical', '▁those', '▁comp', 'huh', '▁live', 'tor', 'ise', '▁old', 'am', 'rr', '▁sta', '▁n', 'ick', 'di', 'ma', 'ary', 'ction', '▁friend', 'ition', '▁gu', '▁through', 'pp', 'for', 'ie', 'ious', '▁sh', '▁home', 'lu', '▁high', 'ian', 'cu', '▁help', '▁give', '▁talk', '▁sha', '▁such', '▁didn', 'em', '▁may', '▁ga', "▁'", '▁gra', '▁guess', '▁every', '▁app', 'tic', '▁tra', '▁"', 'op', '▁made', '"', '▁op', '▁own', '▁mar', 'no', '▁ph', '▁life', '▁y', 'ak', 'ine', '▁pu', '▁place', '▁always', '▁start', '▁jo', '▁pe', '▁let', '▁name', 'ni', '▁same', '▁last', '▁cl', 'ph', '▁both', '▁pri', 'ities', '▁another', 'and', '▁al', '▁boy', 'ving', '▁actually', '▁person', '▁went', '▁yes', 'ca', 'ally', '▁h', '▁great', '▁thought', '▁used', 'act', '▁feel', 'ward', '▁different', '▁cons', '▁show', '▁watch', '▁being', '▁money', 'ay', '▁try', '▁why', '▁big', 'ens', '▁cha', '▁find', '▁hand', '▁real', '▁four', 'ial', '▁ne', '▁che', '▁read', '▁five', '▁family', 'ag', '▁change', '▁add', 'ha', '▁put', 'par', 'lic', 'side', '▁came', '▁under', 'ness', '▁per', 'j', '▁around', '▁end', '▁house', 'if', '▁while', 'vo', '▁act', '▁happen', '▁plan', 'mit', '▁far', '▁tri', '▁ten', '▁du', '▁win', '▁tea', 'ze', '▁better', '▁sure', '▁mu', '▁use', '▁anything', '▁love', '▁world', '▁hard', 'ure', '▁does', '▁war', '▁stuff', '▁ja', '▁must', 'min', 'gg', '▁ru', '▁care', '▁tell', '▁pl', '▁doing', '▁probably', '▁found', 'ative', '▁point', 'ach', '▁ju', 'ip', '▁again', '▁interest', '▁state', '▁week', 'na', '▁might', '▁pretty', '▁ki', '▁fo', 'ber', '▁am', 'line', 'led', '▁six', '▁acc', '▁bri', '▁call', '▁sw', '▁each', '▁business', '▁keep', '▁away', 'cause', '▁pass', '▁va', '▁children', '▁pay', '▁count', '▁public', '▁everything', 'land', '▁though', '▁men', 'bo', '▁young', '▁na', '▁move', 'ough', 'ating', 'com', '▁month', 'ton', '▁close', '▁few', '!', '▁maybe', '▁imp', 'son', '▁grow', '▁u', '▁turn', 'ible', '▁em', '▁air', '▁ever', 'our', '▁sea', '▁fun', '▁government', '▁miss', '▁done', '▁next', '▁kids', '▁cor', '▁set', '▁run', 'way', '▁wa', '▁getting', '▁eight', '▁open', '▁job', '▁problem', 'ook', '▁night', '▁learn', '▁book', 'ual', '▁ti', '▁best', 'cept', '▁during', '▁small', 'ex', '▁without', '▁water', '▁trans', '▁course', '▁once', '▁sit', '▁area', '▁country', '▁mister', '▁nothing', '▁whole', '▁believe', '▁service', '▁took', '▁face', '▁bad', '▁later', '▁head', '▁called', '▁seven', '▁art', '▁since', '▁er', '▁fact', '▁city', '▁market', '▁hour', '▁continue', 'ship', '▁invest', '▁exactly', '▁large', '▁true', '▁nine', '▁sub', '▁having', '▁game', 'va', '▁lu', '▁conf', '▁case', '▁doesn', '▁certain', '▁wi', '▁law', '▁else', 'fi', '▁left', '▁enough', '▁second', '▁gonna', '▁food', '▁hope', '▁saw', '▁between', '▁je', 'bi', '▁girl', '▁company', '▁able', '▁expect', '▁told', '▁stand', '▁group', '▁main', '▁walk', '▁cause', '▁however', '▁number', '▁follow', '▁near', '▁yet', '▁sometimes', '▁train', '▁lead', '▁system', '▁remain', '▁develop', 'gra', '▁word', '▁exc', '▁together', '▁consider', '▁town', '▁less', 'ator', '▁important', '▁remember', '▁free', '▁quite', '▁understand', '▁bra', '▁support', '▁idea', '▁stop', '▁reason', '▁nice', '▁mm', '▁agree', '▁low', '▁against', '▁issue', '▁become', '▁today', '▁side', '▁student', '▁matter', '▁question', '▁mother', '▁father', '▁hundred', '▁sort', '▁eat', '▁already', '▁rest', '▁line', '▁asked', '▁include', '▁upon', '▁office', '▁won', '▁class', '▁wait', '▁twenty', '▁half', '▁light', '▁price', '▁almost', 'ash', '▁child', '▁sign', '▁least', '▁several', 'press', '▁either', '▁minute', '▁himself', '▁parents', '▁room', '▁whatever', '▁general', '▁cost', '▁among', '▁direct', '▁computer', '▁appear', '▁meet', '▁ski', '▁return', '▁couple', '▁product', '▁suppose', '▁definitely', '▁america', '▁term', '▁usually', '▁strong', '▁current', '▁arm', '▁speak', '▁local', '▁south', '▁experience', '▁full', '▁north', '▁elect', '▁leave', '▁provide', 'qui', '▁power', '▁movie', '▁everyone', '▁making', '▁member', '▁woman', '▁somebody', '▁wonder', '▁short', '▁health', '▁police', '▁bank', '▁until', '▁companies', '▁everybody', '▁knew', '▁program', '▁music', '▁york', '▁land', '▁doctor', '▁answer', '▁building', '▁employ', '▁travel', '▁major', '▁seems', '▁safe', 'gue', '▁college', '▁along', '▁clear', '▁especially', '▁umhu', '▁result', '▁type', '▁court', '▁black', '▁hold', '▁myself', '▁education', '▁social', '▁enjoy', '▁became', '▁whether', '▁morning', '▁difficult', '▁shi', '▁felt', '▁husband', '▁white', '▁taking', '▁million', '▁require', '▁early', 'ency', '▁visit', '▁level', '▁brother', '▁married', '▁further', '▁affect', '▁serve', '▁present', '▁park', '▁effect', '▁wife', '▁teacher', '▁cannot', '▁community', '▁street', '▁period', '▁national', '▁view', '▁future', '▁daughter', '▁situation', '▁grand', '▁success', '▁perform', '▁concern', '▁complete', '▁example', 'ized', '▁thousand', '▁increase', '▁began', '▁final', '▁east', '▁sense', '▁charge', '▁record', '▁born', '▁instead', '▁receive', '▁women', '▁across', '▁information', '▁although', '▁process', '▁condition', '▁security', '▁treat', '▁funny', '▁custom', '▁cold', '▁behind', 'ified', '▁ground', 'cycl', '▁depend', '▁themselves', '▁design', '▁slow', '▁third', '▁smoke', '▁wrong', '▁project', '▁space', '▁drink', '▁particular', '▁listen', '▁thirty', '▁special', 'ability', '▁improve', '▁attack', '▁happy', '▁strange', '▁english', '▁value', '▁brought', '▁private', '▁account', '▁china', '▁spoke', '▁foreign', '▁possible', '▁author', '▁circ', '▁voice', '▁figure', '▁control', '▁according', '▁green', '▁university', '▁language', '▁please', '▁animal', '▁church', '▁society', '▁dream', '’', 'q', ':', ';', '—', '‘', '”', '_', '3', '8', '<', '>', '1', '–', '7', '(', ')', '0', '2', '4', '+', '&', '5', '9', 'ü', 'é', '/', 'á', 'ó', 'ō', 'ú', ']', 'â', 'í', 'ã', 'ð', 'ā', 'ć', 'č', 'š', 'è', 'ë', '`', 'ç', 'ū', 'ạ', 'ø', '=', 'à', 'ł', 'α', 'ô', 'к', '}', 'å', 'ă', 'и', 'ī', 'π', 'œ', '\\', '[', 'ñ', 'ß', 'ö', 'ä', '6', 'з', 'н', 'û', '%', '{', '¡', 'æ', 'ê', 'þ', 'ę', 'ě', 'ğ', 'ń', 'ő', 'ř', 'ž', 'ʻ', 'в', 'е', 'й', 'л', 'ь', 'χ', '“']
Now let’s update the vocabulary in this model, using the German tokenizer that we have trained in the data preparation step.
# Lets change the tokenizer vocabulary by passing the path to the new directory,
asr_model.change_vocabulary(
new_tokenizer_dir="../data_preparation/data/processed/tokenizer/tokenizer_spe_bpe_v1024/",
new_tokenizer_type="bpe"
)
[NeMo W 2022-10-25 00:26:44 modelPT:217] You tried to register an artifact under config key=tokenizer.model_path but an artifact for it has already been registered.
[NeMo W 2022-10-25 00:26:44 modelPT:217] You tried to register an artifact under config key=tokenizer.vocab_path but an artifact for it has already been registered.
[NeMo I 2022-10-25 00:26:44 mixins:170] Tokenizer SentencePieceTokenizer initialized with 1024 tokens
[NeMo I 2022-10-25 00:26:44 ctc_bpe_models:259]
Replacing old number of classes (1024) with new number of classes - 1024
[NeMo I 2022-10-25 00:26:45 ctc_bpe_models:301] Changed tokenizer to ['<unk>', 'en', 'er', '▁d', 'ch', 'ei', 'un', 'ie', '▁w', '▁a', '▁s', '▁i', 'st', '▁die', '▁un', '▁m', 'ge', 'ich', '▁da', 'ein', 'ss', '▁b', '▁h', 'sch', '▁v', 'on', 'an', '▁k', '▁z', '▁n', '▁und', 'gen', '▁f', '▁e', 'ir', '▁au', 'ti', '▁ein', '▁der', 'll', 'in', '▁wir', 'te', '▁in', 'or', 'ur', 'ten', '▁ge', 'ung', 'ra', 'it', 're', 'ar', '▁zu', 'den', '▁g', 'der', '▁p', 'al', 'ür', 'lich', 'hr', 'icht', 'es', '▁ha', 'men', '▁das', 'ben', '▁ver', 'eit', 'em', '▁ist', 'ier', '▁den', 'tz', '▁l', 'ber', '▁be', '▁dass', '▁an', '▁auch', 'om', '▁nicht', 'de', '▁es', 'isch', '▁mit', 'ter', 'se', '▁ich', 'au', 'op', '▁er', '▁t', 'oll', 'ach', '▁j', '▁eur', 'ig', 'um', '▁für', '▁auf', '▁europ', '▁sie', 'ol', '▁sch', '▁re', 'tion', 'ro', '▁haben', 'ft', 'nen', '▁im', 'ion', 'ssen', '▁von', 'ehr', '▁eine', 'la', 'be', '▁st', 'sp', '▁wer', '▁en', '▁dies', 'rei', '▁vor', '▁uns', 'eh', 'el', 'is', 'eu', 'at', '▁kom', 'ungen', 'sa', 'and', 'über', 'ischen', 'ind', '▁europä', 'chen', '▁des', '▁aus', 'cht', 'iss', '▁werden', '▁um', 'ck', '▁sich', '▁o', 'tsch', 'iel', 'fen', '▁zw', 'än', '▁her', '▁bei', '▁all', '▁sind', '▁wie', 'rau', '▁al', 'as', '▁aber', 'anz', '▁so', '▁dem', 'uss', 'iti', '▁wen', '▁über', 'kt', '▁hat', 'och', '▁komm', 'tw', 'rä', 'etz', 'tig', '▁als', 'hal', 'ah', '▁r', 'aft', 'ahr', '▁hier', 'eil', '▁wenn', 'si', '▁ab', 'ger', 'ien', 'ste', 'il', '▁unter', 'eg', 'zu', '▁kommiss', 'lie', 'sten', 'schen', 'ann', '▁wird', '▁mü', 'ation', 'oliti', 'ment', '▁dan', '▁herr', '▁ber', 'ri', 'ru', 'sen', 'gt', '▁europäischen', 'ungs', 'ne', 'ische', 'sam', 'egen', '▁mehr', '▁nach', 'ön', 'ver', '▁müssen', 'heit', '▁man', 'us', 'eine', '▁diese', 'lei', '▁kon', 'dern', 'aten', 'ün', '▁par', '▁zwei', 'hen', '▁pro', '▁gr', 'tlich', 'sta', '▁ar', 'ere', 'räsi', 'räsiden', '▁kön', 'ffen', '▁noch', 'im', 'är', 'ht', '▁nur', '▁koll', 'ken', '▁ra', 'setz', 'ern', '▁was', '▁soll', '▁viel', 'und', '▁kann', '▁einen', '▁ih', 'ommen', '▁kommission', '▁dann', 'ät', 'mit', '▁muss', '▁war', 'llen', '▁men', 'bt', 'chte', '▁ste', 'ieren', '▁sehr', 'mer', 'eren', 'keit', '▁fra', '▁kl', 'beit', 'urch', '▁sein', '▁mö', 'ort', '▁präsiden', 'tschaft', '▁können', '▁dar', '▁parla', 'etzt', 'innen', 'glie', '▁eu', '▁menschen', 'rie', 'gel', '▁zur', '▁mitglie', 'ut', '▁parlament', '▁europa', '▁durch', '▁einer', 'ill', '▁bür', 'utz', 'ör', '▁ander', '▁gem', 'le', '▁jetzt', '▁mitglied', '▁oder', 'zi', 'rit', 'olitik', '▁fin', 'kommen', 'wer', 'ühr', 'ff', 'lau', '▁sa', '▁zum', 'staaten', 'igen', '▁dieser', 'ang', 'ick', 'eiten', 'for', 'ok', 'ande', '▁ganz', 'lichen', 'tra', 'ierung', '▁union', '▁gen', 'ahl', 'ehmen', 'ßen', '▁sp', '▁reg', '▁gegen', 'lo', 'ichtig', '▁europäische', 'ück', '▁hin', 'ollen', '▁weit', '▁bürger', 'iert', '▁fa', '▁gi', 'du', '▁neu', '▁recht', '▁mach', 'ktion', 'eigen', '▁mitgliedstaaten', '▁ger', 'her', '▁wo', 'ahren', 'andel', 'tel', '▁gro', 'est', '▁gew', '▁alle', 'schaft', 'liche', 'enz', 'ord', '▁denn', 'ität', '▁son', 'ul', '▁for', 'mal', 'che', 'rin', 'tet', '▁inter', '▁wollen', 'ünf', '▁gibt', 'ab', 'bar', 'ran', '▁gel', 'dert', 'tiv', '▁diesem', '▁heu', '▁mein', '▁damit', '▁geht', '▁schon', '▁frau', 'land', 'elt', 'tige', '▁brau', '▁dieses', '▁ja', '▁bis', '▁ent', '▁wirk', 'geb', 'glich', '▁de', 'okra', 'stim', '▁entw', 'sammen', '▁einem', 'kte', '▁deu', '▁rat', 'wei', '▁weil', 'ke', 'ehn', '▁ihr', 'halb', 'dig', 'ing', 'ütz', '▁immer', 'tigen', '▁gemein', 'hl', 'ausen', '▁heute', 'ass', '▁bet', '▁eigen', '▁keine', 'pf', 'alen', '▁haus', '▁entsch', '▁sicher', '▁kollegen', 'aus', '▁glau', '▁ihre', '▁verh', 'me', '▁zusammen', '▁präsident', '▁ö', 'gr', 'für', 'ige', '▁arbeit', '▁gef', '▁ch', 'sche', '▁sondern', 'ik', 'stell', '▁sta', '▁je', '▁tausen', 'ähr', '▁besch', 'bl', 'rechen', '▁finanz', '▁entwick', '▁wieder', 'ale', '▁wi', '▁mich', '▁verb', '▁tausend', 'lä', '▁ko', '▁sagen', '▁wirklich', '▁groß', '▁möchte', '▁wichtig', 'iz', '▁unsere', '▁wirtschaft', '▁brauchen', 'äm', 'ist', 'gie', 'ehen', 'dlich', 'leich', 'änder', 'nehmen', '▁machen', 'äch', '▁lie', 'eich', '▁kommissar', 'he', '▁land', 'ga', 'ner', '▁la', '▁eben', '▁gesch', 'tin', '▁ob', 'ssch', '▁ange', 'lle', 'halt', 'ionen', '▁doch', 'nung', '▁dafür', '▁frage', '▁unser', 'wir', '▁habe', 'nah', '▁kein', 'ert', '▁le', '▁unterst', 'hör', '▁ma', '▁nat', '▁am', '▁sozi', '▁vert', '▁drei', 'orden', '▁sollten', '▁kolleg', 'stän', '▁lei', '▁wur', '▁weiter', 'gra', 'eson', 'stimm', 'esonder', '▁fünf', '▁unterstütz', '▁erf', '▁politi', '▁grund', '▁letz', '▁deshalb', '▁diesen', 'ze', 'gan', 'ade', 'teil', '▁meine', 'dung', 'schla', 'andl', '▁schw', '▁stra', 'bei', 'fach', 'undert', 'rü', '▁sel', 'lin', '▁wür', 'arbeit', 'arkt', 'gend', '▁bericht', 'schie', 'halten', 'ürlich', '▁entschei', '▁gu', 'äre', 'ta', 'kun', '▁verf', '▁einmal', '▁anderen', 'vor', 'führ', '▁ins', '▁natürlich', 'zig', 'blem', '▁entwickl', '▁möglich', '▁etw', '▁ihnen', '▁sol', '▁mill', 'ats', 'spiel', 'wegen', 'ritt', '▁glaube', 'af', '▁demokra', '▁hundert', '▁kolleginnen', 'ech', '▁ziel', 'tt', 'affen', '▁jahr', '▁darau', '▁mir', '▁vier', '▁regel', '▁problem', 'alt', 'bst', '▁viele', '▁gerade', 'isk', '▁ho', 'stru', '▁will', 'qu', '▁gut', '▁klar', '▁komp', 'fa', 'acht', '▁c', 'ers', 'tieren', '▁handel', 'fä', '▁wissen', '▁dazu', 'itu', 'spar', '▁tun', '▁weg', '▁leben', 'eiß', '▁nation', 'os', 'pp', 'nis', '▁frei', 'dem', 'trag', 'zehn', '▁wel', 'kunft', 'igkeit', 'setzen', '▁te', '▁teil', '▁zeit', '▁gemeinsam', '▁jahren', '▁deswegen', 'zen', '▁min', 'gehen', 'pt', 'det', 'zie', '▁ex', '▁bed', '▁also', '▁ener', '▁darauf', 'su', '▁beispiel', 'sagt', '▁bereit', '▁we', 'spro', '▁disk', '▁selbst', 'ition', '▁abge', '▁aller', 'og', 'ker', '▁ne', 'orgen', 'schutz', 'rom', 'anzig', 'sicht', '▁rechts', 'bil', 'tes', 'kehr', 'recht', '▁fest', '▁richt', '▁deutsch', '▁verhandl', 'lan', 'ster', 'reich', '▁neue', '▁zwanzig', 'am', 'end', '▁sy', '▁zukunft', 'äu', '▁no', '▁gleich', '▁ta', '▁dort', '▁umw', '▁tran', '▁führ', '▁fraktion', 'ma', '▁mo', 'stellen', '▁intere', 'antw', 'form', '▁best', 'politik', 'sschuss', 'hne', 'ätz', 'nahmen', '▁grenz', 'ktur', 'stem', '▁geb', 'strie', '▁bin', 'stellt', '▁ausge', '▁gesetz', '▁haushalt', '▁zwischen', 'ös', '▁wurde', '▁worden', 'sprechen', 'esen', 'sser', 'chten', '▁alles', '▁etwas', '▁gesagt', '▁sicherheit', 'ds', '▁wäre', 'ringen', 'schlag', '▁kommen', 'öl', 'ahre', 'sell', '▁mal', '▁geld', '▁einfach', 'fe', 'tie', 'erst', 'weise', 'bat', 'reichen', '▁mittel', 'ekt', 'hem', 'ima', 'ischer', 'mittel', '▁letzten', '▁politik', 'kl', 'hn', 'ktiv', '▁ohne', '▁geben', '▁besonder', 'äl', 'cher', '▁hät', '▁län', '▁allen', '▁ihrer', 'mm', 'inn', '▁würde', '▁deutlich', 'id', 'iger', '▁sei', '▁fl', '▁inv', 'rechte', 'spolitik', '▁bek', '▁eines', '▁klima', 'ordnung', '▁eigentlich', '▁unternehmen', 'tr', 'wie', '▁ern', '▁maß', '▁einge', '▁liebe', 'fl', 'ativ', 'geben', '▁stär', '▁welt', 'wo', 'uer', 'zei', 'tens', '▁tür', 'ahlen', 'sse', 'tur', '▁bit', 'ffentlich', '▁präsidentin', 'iv', 'satz', '▁nun', '▁euro', '▁produ', 'sk', 'att', 'chn', '▁sti', '▁allem', '▁umwelt', 'gang', 'hmen', 'olle', '▁russ', '▁jahre', 'ichtige', '▁invest', '▁energie', 'iell', '▁genau', '▁endlich', 'anken', '▁blei', '▁gest', '▁intern', 'min', '▁kr', '▁pa', 'hema', '▁fre', 'tisch', '▁part', '▁debat', '▁andere', 'fin', 'ruck', '▁gar', '▁sozial', 'hin', 'markt', 'gangen', '▁dabei', 'unden', '▁indu', '▁star', '▁bereich', '▁unserer', '▁entwicklung', '▁große', '▁thema', 'ehl', '▁verantw', '▁abkommen', 'ffe', '▁darüber', 'di', 'hau', 'olg', 'ange', 'stand', '▁notw', '▁türk', '▁neuen', '▁nichts', 'ituation', '▁em', 'setzt', '▁waren', '▁gemacht', 'sellschaft', 'ot', 'alis', 'ding', '▁dür', '▁chin', '▁heraus', '▁verhandlungen', '▁ges', '▁näm', '▁seit', '▁nämlich', '▁öffentlich', 'kre', 'isse', 'unkt', 'keiten', '▁komprom', '▁internation', 'pro', 'ander', '▁pers', 'wirtschaft', 'fra', 'tre', 'wert', '▁industrie', '▁di', '▁erw', '▁heiß', '▁millionen', '▁bürgerinnen', '▁wei', '▁herren', '▁sollte', 'auen', 'eicht', 'beiten', '▁gehör', '▁verst', '▁lassen', 'rise', '▁fol', '▁kin', '▁darf', '▁sech', '▁heißt', '▁bisher', 'mus', '▁kor', '▁rei', 'ha', '▁', 'e', 'n', 'i', 'r', 's', 't', 'a', 'd', 'h', 'u', 'l', 'g', 'c', 'm', 'o', 'b', 'w', 'f', 'k', ',', 'z', 'p', 'v', '.', 'ü', 'ä', 'ö', 'j', 'ß', '!', 'y', '?', 'x', 'q', '„', '2', '0', 'ğ', 'á', '1', '—', 'č', 'š', 'é', '5', '3', '4', '7', '8', "'", 'ï', 'ń', '9', 'è', 'ó', 'ć', '°', 'ç', 'í', 'ñ', 'ľ', 'ň', 'ş'] vocabulary.
After this, our decoder has completely changed, but our encoder (where most of the weights are) remained intact.
Update Config#
Each NeMo model has a config embedded in it, which can be accessed via model.cfg. In general, this is the config that was used to construct the model.
For pre-trained models, this config generally represents the config used to construct the model when it was trained. A nice benefit to this embedded config is that we can repurpose it to set up new data loaders, optimizers, schedulers, and even data augmentation!
!ln -s ../data_preparation/data .
ln: failed to create symbolic link './data': File exists
DATA_ROOT = "./data"
USE_TARRED_DATASET = True
if USE_TARRED_DATASET:
# Setup train, validation, test configs
with open_dict(asr_model.cfg):
# Train dataset (Concatenate train manifest cleaned and dev manifest cleaned)
asr_model.cfg.train_ds.manifest_filepath = f'{DATA_ROOT}/processed/tar/train/tarred_audio_manifest.json'
asr_model.cfg.train_ds.is_tarred = True
asr_model.cfg.train_ds.tarred_audio_filepaths= DATA_ROOT+'/processed/tar/train/audio_{0..127}.tar'
asr_model.cfg.train_ds.batch_size = 32
asr_model.cfg.train_ds.num_workers = 32
asr_model.cfg.train_ds.pin_memory = True
asr_model.cfg.train_ds.trim_silence = True
# Validation dataset (Use test dataset as validation, since we train using train + dev)
asr_model.cfg.validation_ds.manifest_filepath = [f'{DATA_ROOT}/processed/test_manifest_merged.json', f'{DATA_ROOT}/processed/dev_manifest_merged.json']
asr_model.cfg.validation_ds.batch_size = 32
asr_model.cfg.validation_ds.num_workers = 32
asr_model.cfg.validation_ds.pin_memory = True
asr_model.cfg.validation_ds.trim_silence = True
else:
# Setup train, validation, test configs
with open_dict(asr_model.cfg):
# Train dataset (Concatenate train manifest cleaned and dev manifest cleaned)
asr_model.cfg.train_ds.manifest_filepath = f'{DATA_ROOT}/processed/train_manifest_merged.json'
asr_model.cfg.train_ds.batch_size = 32
asr_model.cfg.train_ds.num_workers = 32
asr_model.cfg.train_ds.pin_memory = True
asr_model.cfg.train_ds.trim_silence = True
# Validation dataset (Use test dataset as validation, since we train using train + dev)
asr_model.cfg.validation_ds.manifest_filepath = [f'{DATA_ROOT}/processed/test_manifest_merged.json', f'{DATA_ROOT}/processed/dev_manifest_merged.json']
asr_model.cfg.validation_ds.batch_size = 32
asr_model.cfg.validation_ds.num_workers = 32
asr_model.cfg.validation_ds.pin_memory = True
asr_model.cfg.validation_ds.trim_silence = True
# Point to the new train and validation data for fine-tuning
asr_model.setup_training_data(train_data_config=asr_model.cfg.train_ds)
asr_model.setup_validation_data(val_data_config=asr_model.cfg.validation_ds)
[NeMo W 2022-10-25 00:26:45 audio_to_text_dataset:179] dataset does not have explicitly defined labels
[NeMo I 2022-10-25 00:26:46 collections:194] Dataset loaded with 9029 files totalling 18.10 hours
[NeMo I 2022-10-25 00:26:46 collections:195] 427 files were filtered totalling 2.17 hours
[NeMo W 2022-10-25 00:26:46 ctc_models:434] Model Trainer was not set before constructing the dataset, incorrect number of training batches will be used. Please set the trainer and rebuild the dataset.
[NeMo I 2022-10-25 00:26:51 collections:194] Dataset loaded with 4077 files totalling 9.90 hours
[NeMo I 2022-10-25 00:26:51 collections:195] 0 files were filtered totalling 0.00 hours
Setting up optimizer and scheduler#
When fine-tuning character models, it is generally advised to use a lower learning rate and reduced warmup. A reduced learning rate helps preserve the pre-trained weights of the encoder. Since the fine-tuning dataset is generally smaller than the original training dataset, the warmup steps would be far too much for the smaller fine-tuning dataset.
# Original optimizer + scheduler
print(OmegaConf.to_yaml(asr_model.cfg.optim))
name: novograd
lr: 0.05
betas:
- 0.8
- 0.25
weight_decay: 0.001
sched:
name: CosineAnnealing
warmup_steps: 1000
warmup_ratio: null
min_lr: 1.0e-05
last_epoch: -1
# Use the smaller learning rate we set before
with open_dict(asr_model.cfg.optim):
asr_model.cfg.optim.name="adamw"
asr_model.cfg.optim.lr = 0.01
asr_model.cfg.optim.betas = [0.8, 0.25] # from paper
asr_model.cfg.optim.weight_decay = 0.001 # Original weight decay
asr_model.cfg.optim.sched.warmup_steps = None # Remove default number of steps of warmup
asr_model.cfg.optim.sched.warmup_ratio = 0.05 # 5 % warmup
asr_model.cfg.optim.sched.min_lr = 1e-5
asr_model.cfg.optim.sched.max_steps = 50000
Training the model#
And now we can create a PyTorch Lightning trainer and call fit. To increase training speed, we can leverage the mixed precision training mode. In this notebook, we demonstrate training with 1 GPUs. To train with 8 GPUs, execute the train.py script in a shell terminal.
Notes:
Even with cross-language transfer learning, the model will still take a few hundreds epochs to train to convergence.
To stabilize training and avoid NAN loss issues, increase the global batch size to the range of [256, 2048]. On devices with small memory, this can be achieved by setting an appropriate number of the
accumulate_grad_batches.asr_model.cfg.train_ds.batch_sizedenotes the per-device batchsize. The global batch size will bebatch_size* #nodes * GPUs per node * accumulate_grad_batches.
from pytorch_lightning.callbacks import ModelCheckpoint
import pytorch_lightning as pl
checkpoint_callback = ModelCheckpoint(
save_top_k=10,
monitor="val_wer",
mode="min",
dirpath="./checkpoint-dir",
filename="citrinet-DE-{epoch:02d}",
save_on_train_epoch_end=True,
)
trainer = pl.Trainer(precision=16,
devices=1,
accelerator='gpu',
max_epochs=500,
default_root_dir="./checkpoint/",
accumulate_grad_batches=32, # For a global batch size of 32*1*32 = 1024
callbacks=[checkpoint_callback])
trainer.fit(asr_model)
asr_model.save_to('de-asr-model.nemo')