ASR ---------------------- In the simplest use case, you can deploy an ASR model without any language model as follows: .. code-block:: bash :substitutions: jarvis-build speech_recognition \ /servicemaker-dev/: \ /servicemaker-dev/: \ --acoustic_model_name= where: - ```` is the encryption key used during the export of the ``.ejrvs`` file. - ```` is the name is the acoustic model, which will be used to name the Jarvis ASR model in Triton. - ```` is the name of the ``ejrvs`` file to use as input. - ```` is the Jarvis ``jmir`` file that will be generated. Upon succesful completion of this command, a file named ```` will be created in the ``/servicemaker-dev/`` folder. Since no language model is specified, the Jarvis greedy decoder will be used to predict the transcript based on the output of the acoustic model. Language Models ^^^^^^^^^^^^^^^^^^^ Jarvis ASR supports decoding with an n-gram language model. The n-gram language model can be stored in a ``.arpa`` format or a KenLM binary format. To prepare the Jarvis JMIR configuration using an n-gram language model stored in ``arpa`` format, run: .. code-block:: bash :substitutions: jarvis-build speech_recognition \ /servicemaker-dev/: \ /servicemaker-dev/: \ --acoustic_model_name= \ --decoding_language_model_arpa= To use Jarvis ASR with a KenLM binary file, generate the Jarvis JMIR with: .. code-block:: bash :substitutions: jarvis-build speech_recognition \ /servicemaker-dev/: \ /servicemaker-dev/: \ --acoustic_model_name= \ --decoding_language_model_binary= The decoder language model hyper-parameters (``alpha``, ``beta``, and ``beam_search_width``) can also be set from the ``jarvis-build`` command. .. code-block:: bash :substitutions: jarvis-build speech_recognition \ /servicemaker-dev/: \ /servicemaker-dev/: \ --acoustic_model_name= \ --decoding_language_model_binary= \ --lm_decoder_cpu.beam_search_width= \ --lm_decoder_cpu.language_model_alpha= \ --lm_decoder_cpu.language_model_beta= Streaming/Offline Configuration ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ By default, the Jarvis JMIR file is configured to be used with the Jarvis StreamingRecognize RPC call, for streaming use cases. To use the StreamingRecognize RPC call, generate the Jarvis JMIR file by adding the ``--offline`` option. .. code-block:: bash :substitutions: jarvis-build speech_recognition \ /servicemaker-dev/: \ /servicemaker-dev/: \ --acoustic_model_name= \ --offline Furthermore, the default streaming Jarvis JMIR configuration is to provide intermediate transcripts with very low latency. For use cases where being able to support additional concurrent audio streams is more important, run: .. code-block:: bash :substitutions: jarvis-build speech_recognition \ /servicemaker-dev/: \ /servicemaker-dev/: \ --acoustic_model_name= \ --chunk_size=0.8 \ --padding_factor=2 \ --padding_size=0.8 GPU-accelerated Decoder ^^^^^^^^^^^^^^^^^^^^^^^ The Jarvis ASR pipeline can also use a GPU-accelerated weighted finite-state transducer (WFST) decoder that was initially developed for `Kaldi `_. To use the GPU decoder, using a language model defined by an ``.arpa`` file, run: .. code-block:: bash :substitutions: jarvis-build speech_recognition \ /servicemaker-dev/: \ /servicemaker-dev/: \ --acoustic_model_name= \ --decoding_language_model_arpa= \ --gpu_decoder where ```` is the language model ``.arpa`` file that was used during the WFST decoding phase. **Note: Conversion from an .arpa file to a WFST graph can take a long time, especially for large language models.** Also, large language models will increase GPU memory utilization. When using the GPU decoder, it is recommended to use different language models for the WFST decoding phase and the lattice rescoring phase. This can be achieved by using the following ``jarvis-build`` command: .. code-block:: bash :substitutions: jarvis-build speech_recognition \ /servicemaker-dev/: \ /servicemaker-dev/: \ --acoustic_model_name= \ --decoding_language_model_arpa= \ --rescoring_language_model_arpa= \ --gpu_decoder where: - ```` is the language model ``.arpa`` file that was used during the WFST decoding phase. - ```` is the language model used during the lattice rescoring phase. Typically, one would use a small language model for the WFST decoding phase (for example, a pruned 2 or 3-gram language model) and a larger language model for the lattice rescoring phase (for example, an unpruned 4-gram language model). For advanced users, it is also possible to configure the GPU decoder by specifying the decoding WFST file and the vocabulary directly, instead of using an ``.arpa`` file. For example: .. code-block:: bash :substitutions: jarvis-build speech_recognition \ /servicemaker-dev/: \ /servicemaker-dev/: \ --acoustic_model_name= \ --decoding_language_model_fst= \ --decoding_language_model_words= \ --gpu_decoder Furthermore, you can specify the ``.carpa`` files to use in the case where lattice rescoring is needed: .. code-block:: bash :substitutions: jarvis-build speech_recognition \ /servicemaker-dev/: \ /servicemaker-dev/: \ --acoustic_model_name= \ --decoding_language_model_fst= \ --decoding_language_model_carpa= \ --decoding_language_model_words= \ --rescoring_language_model_carpa= \ --gpu_decoder where: - ```` is the language model construct arpa representation to use during the WFST decoding phase. - ```` is the language model construct arpa representation to use during the lattice rescoring phase. The GPU decoder hyper-parameters (``default_beam``, ``lattice_beam``, ``word_insertion_penalty`` and ``acoustis_scale``) can be set with the ``jarvis-build`` command as follows: .. code-block:: bash :substitutions: jarvis-build speech_recognition \ /servicemaker-dev/: \ /servicemaker-dev/: \ --acoustic_model_name= \ --decoding_language_model_arpa= \ --lattice_beam= \ --lm_decoder_gpu.default_beam= \ --lm_decoder_gpu.acoustic_scale= --rescorer.word_insertion_penalty= \ --gpu_decoder Beginning/End of Utterance Detection ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jarvis ASR uses an algorithm that detects the beginning and end of utterances. This algorithm is used to reset the ASR decoder state, and to trigger a call to the punctuator model. By default, the beginning of an utterance is flagged when 20% of the frames in a 300ms window have non-blank characters, and the end of an utterance is flagged when 98% of the frames in a 800ms window are blank characters. Users can tune those values for their particular use case by using the following ``jarvis-build`` command: .. code-block:: bash :substitutions: jarvis-build speech_recognition \ /servicemaker-dev/: \ /servicemaker-dev/: \ --acoustic_model_name= \ --vad.vad_start_history=300 \ --vad.vad_start_th=0.2 \ --vad.vad_stop_history=800 \ --vad.vad_stop_th=0.98 Additionally, it is possible to disable the beginning/end of utterance detection with the following code: .. code-block:: bash :substitutions: jarvis-build speech_recognition \ /servicemaker-dev/: \ /servicemaker-dev/: \ --acoustic_model_name= \ --vad.vad_type=none Note that in this case, the decoder state would only get reset once the full audio signal has been sent by the client. Similarly, the punctuator model would only get called once. Mandarin ^^^^^^^^^ The default parameters values that can be provided to the ``jarvis-build`` command will give accurate transcripts for most use cases. However, for some languages like Mandarin, those parameters values must be tuned. When transcribing Mandarin, the recommended values are: **For streaming recognition** .. code-block:: bash :substitutions: jarvis-build speech_recognition \ /servicemaker-dev/: \ /servicemaker-dev/: \ --acoustic_model_name= \ --chunk_size=1.6 \ --padding_size=3.2 \ --padding_factor=4 \ --vad.vad_stop_history=1600 \ --vad.vad_start_history=200 \ --vad.vad_start_th=0.1 **For offline recognition** .. code-block:: bash :substitutions: jarvis-build speech_recognition \ /servicemaker-dev/: \ /servicemaker-dev/: \ --acoustic_model_name= \ --padding_size=3.2 \ --padding_factor=2 \ --vad.vad_stop_history=1600 \ --vad.vad_start_history=200 \ --vad.vad_start_th=0.1