How do I use Riva TTS APIs with out-of-the-box models?
Contents
![](http://developer.download.nvidia.com/notebooks/dlsw-notebooks/riva_tts_tts-python-basics/nvidia_logo.png)
How do I use Riva TTS APIs with out-of-the-box models?#
This tutorial walks you through the basics of Riva’s TTS services, specifically covering how to use Riva TTS APIs with OOTB (out-of-the-box) models.
NVIDIA Riva Overview#
NVIDIA Riva is a GPU-accelerated SDK for building Speech AI applications that are customized for your use case and deliver real-time performance.
Riva offers a rich set of speech and natural language understanding services such as:
Automated speech recognition (ASR)
Text-to-Speech synthesis (TTS)
A collection of natural language processing (NLP) services, such as named entity recognition (NER), punctuation, and intent classification.
In this tutorial, we will interact with the text-to-speech synthesis (TTS) APIs and customize Riva TTS audio output with SSML.
For more information about Riva, please refer to the Riva developer website.
Basics: Generating Speech with Riva TTS APIs#
The Riva TTS service is based on a two-stage pipeline: Riva generates a mel spectrogram using the first model, then uses the mel spectrogram to generate speech using the second model. This pipeline forms a text-to-speech system that enables you to synthesize natural sounding speech from raw transcripts without any additional information such as patterns or rhythms of speech.
Riva provides two state-of-the-art voices (one male and one female) for English, that can easily be deployed with the Riva Quick Start scripts. Riva also supports easy customization of TTS in various ways, to meet your specific needs.
Subsequent Riva releases will include features such as model registration to support multiple languages/voices with the same API and support for resampling to alternative sampling rates.
Refer to the Riva TTS documentation for more information.
Now, let’s generate audio using Riva APIs with an OOTB English TTS pipeline.
Requirements and setup#
Start the Riva server. Follow the instructions in the Riva Quick Start Guide to deploy OOTB TTS models on the Riva server before running this tutorial. By default, only the English models are deployed.
Install the additional Python libraries to run this tutorial. Run the following commands to install the libraries:
# We need numpy to read the output from the Riva TTS request.
!pip install numpy
!pip install nvidia-riva-client
Import Riva client libraries#
We first import some required libraries, including the Riva client libraries.
import numpy as np
import IPython.display as ipd
import riva.client
Create Riva clients and connect to the Riva server#
The following URI assumes a local deployment of the Riva server on the default port. In case the server deployment is on a different host or via Helm chart on Kubernetes, use an appropriate URI.
auth = riva.client.Auth(uri='localhost:50051')
riva_tts = riva.client.SpeechSynthesisService(auth)
TTS modes#
Riva TTS supports both streaming and batch inference modes. In batch mode, audio is not returned until the full audio sequence for the requested text is generated and can achieve higher throughput. But when making a streaming request, audio chunks are returned as soon as they are generated, significantly reducing the latency (as measured by time to first audio) for large requests.
Setup the TTS API parameters#
sample_rate_hz = 44100
req = {
"language_code" : "en-US",
"encoding" : riva.client.AudioEncoding.LINEAR_PCM , # Currently only LINEAR_PCM is supported
"sample_rate_hz" : sample_rate_hz, # Generate 44.1KHz audio
"voice_name" : "English-US.Female-1" # The name of the voice to generate
}
Understanding TTS API parameters#
Riva TTS supports a number of options while making a text-to-speech request to the gRPC endpoint, as shown above. Let’s learn more about these parameters:
language_code
- Language of the generated audio.en-US
represents English (US) and is currently the only language supported OOTB.encoding
- Type of audio encoding to generate. Currently, onlyLINEAR_PCM
is supported.sample_rate_hz
- Sample rate of the generated audio. Depends on the microphone and is usually22khz
or44khz
.voice_name
- Voice used to synthesize the audio. Currently, Riva offers two OOTB voices (English-US.Female-1
,English-US.Male-1
).
Make a gRPC request to the Riva server#
For batch inference mode, use synthesize
. Results are returned when the entire audio is synthesized.
req["text"] = "Is it recognize speech or wreck a nice beach?"
resp = riva_tts.synthesize(**req)
audio_samples = np.frombuffer(resp.audio, dtype=np.int16)
ipd.Audio(audio_samples, rate=sample_rate_hz)
For online inference, use synthesize_online
. Results are returned in chunks as they are synthesized.
req["text"] = "Is it recognize speech or wreck a nice beach?"
resp = riva_tts.synthesize_online(**req)
empty = np.array([])
for i, rep in enumerate(resp):
audio_samples = np.frombuffer(rep.audio, dtype=np.int16) / (2**15)
print("Chunk: ",i)
ipd.display(ipd.Audio(audio_samples, rate=44100))
empty = np.concatenate((empty, audio_samples))
print("Final synthesis:")
ipd.display(ipd.Audio(empty, rate=44100))
Customizing Riva TTS audio output with SSML#
Speech Synthesis Markup Language (SSML) specification is a markup for directing the performance of the virtual speaker. Riva supports portions of SSML, allowing you to adjust pitch, rate, and pronunciation of the generated audio. SSML support is available only for the FastPitch model at this time. The FastPitch model must be exported using NeMo>=1.5.1 and the nemo2riva>=1.8.0 tool.
All SSML inputs must be a valid XML document and use the
Riva TTS supports the following SSML tags:
The
prosody
tag, which supports attributesrate
,pitch
, andvolume
, through which we can control the rate, pitch, and volume of the generated audio.The
phoneme
tag, which allows us to control the pronunciation of the generated audio.The
sub
tag, which allows us to replace the pronounciation of the specified word or phrase with a different word or phrase.
Let’s look at customization of Riva TTS with these SSML tags in some detail.
Customizing rate, pitch, and volume with the prosody
tag#
Pitch Attribute#
Riva supports an additive relative change to the pitch. The pitch
attribute has a range of [-3, 3]. Values outside this range result in an error being logged and no audio returned. This value returns a pitch shift of the attribute value multiplied with the speaker’s pitch standard deviation when the FastPitch model is trained. For the pretrained checkpoint that was trained on LJSpeech, the standard deviation was 52.185. For example, a pitch shift of 1.25 results in a change of 1.25*52.185=~65.23Hz pitch shift up.
Riva also supports the prosody tags as per the SSML specs. Prosody tags x-low
, low
, medium
, high
, x-high
, and default
are supported.
The pitch
attribute is expressed in the following formats:
pitch="1"
pitch="+1.8"
pitch="-0.65"
pitch="high"
pitch="default"
For the pretrained Female-1 checkpoint, the standard deviation is 53.33 Hz. For the pretrained Male-1 checkpoint, the standard deviation is 47.15 Hz.
The pitch
attribute does not support Hz
, st
, and %
changes. Support is planned for a future Riva release.
Rate Attribute#
Riva supports a percentage relative change to the rate. The rate
attribute has a range of [25%, 250%]. Values outside this range result in an error being logged and no audio returned.
Riva also supports the prosody tags as per the SSML specs. Prosody tags x-low
, low
, medium
, high
, x-high
, and default
are supported.
The rate
attribute is expressed in the following formats:
rate="35%"
rate="+200%"
rate="low"
rate="default"
Volume Attribute#
Riva supports the volume attribute as described in the SSML specs. The volume attribute supports a range of [-13, 8]dB. Values outside this range result in an error being logged and no audio returned. Tags silent
, x-soft
, soft
, medium
, loud
, x-loud
, and default
are supported.
The volume attribute is expressed in the following formats:
volume="+1dB"
volume="-5.7dB"
volume="x-loud"
volume="default"
Let’s look at an example showing the pitch, rate, and volume customizations for Riva TTS:
"""
Raw text is "Today is a sunny day. But it might rain tomorrow."
We are updating this raw text with SSML:
1. Envelope raw text in '<speak>' tags as is required for SSML
2. Add '<prosody>' tag with 'pitch' attribute set to '2.5'
3. Add '<prosody>' tag with 'rate' attribute set to 'high'
4. Add '<volume>' tag with 'volume' attribute set to '+1dB'
"""
raw_text = "Today is a sunny day. But it might rain tomorrow."
ssml_text = """<speak><prosody pitch='2.5'>Today is a sunny day</prosody>. <prosody rate='high' volume='+1dB'>But it might rain tomorrow.</prosody></speak>"""
print("Raw Text: ", raw_text)
print("SSML Text: ", ssml_text)
req["text"] = ssml_text
# Request to Riva TTS to synthesize audio
resp = riva_tts.synthesize(**req)
# Playing the generated audio from Riva TTS request
audio_samples = np.frombuffer(resp.audio, dtype=np.int16)
ipd.display(ipd.Audio(audio_samples, rate=sample_rate_hz))
Expected results if you run the tutorial:#
<prosody pitch='2.5'>Today is a sunny day</prosody>. <prosody rate='high' volume='+1dB'>But it might rain tomorrow.</prosody>
Note#
If the audio controls are not seen throughout notebook. Open the notebook in github dev or view it in the riva docs
Here are more examples showing the effects of changes in pitch and rate attribute values on the generated audio:
# SSML texts we want to try
ssml_texts = [
"""<speak>This is a normal sentence</speak>""",
"""<speak><prosody pitch="0." rate="100%">This is also a normal sentence</prosody></speak>""",
"""<speak><prosody rate="200%">This is a fast sentence</prosody></speak>""",
"""<speak><prosody rate="60%">This is a slow sentence</prosody></speak>""",
"""<speak><prosody pitch="+1.0">Now, I'm speaking a bit higher</prosody></speak>""",
"""<speak><prosody pitch="-0.5">And now, I'm speaking a bit lower</prosody></speak>""",
"""<speak>S S M L supports <prosody pitch="-1">nested tags. So I can speak <prosody rate="150%">faster</prosody>, <prosody rate="75%">or slower</prosody>, as desired.</prosody></speak>""",
"""<speak><prosody volume='x-soft'>I'm speaking softly.</prosody><prosody volume='x-loud'> And now, This is loud.</prosody></speak>""",
]
# Loop through 'ssml_texts' list and synthesize audio with Riva TTS for each entry 'ssml_texts'
for ssml_text in ssml_texts:
req["text"] = ssml_text
resp = riva_tts.synthesize(**req)
audio_samples = np.frombuffer(resp.audio, dtype=np.int16)
print("SSML Text: ", ssml_text)
ipd.display(ipd.Audio(audio_samples, rate=sample_rate_hz))
print("--------------------------------------------")
Expected results if you run the tutorial:#
This is a normal sentence
<prosody pitch="0." rate="100%">This is also a normal sentence</prosody>
<prosody rate="200%">This is a fast sentence</prosody>
<prosody rate="60%">This is a slow sentence</prosody>
<prosody pitch="+1.0">Now, I'm speaking a bit higher</prosody>
<prosody pitch="-0.5">And now, I'm speaking a bit lower</prosody>
S S M L supports <prosody pitch="-1">nested tags. So I can speak <prosody rate="150%">faster</prosody>, <prosody rate="75%">or slower</prosody>, as desired.</prosody>
<prosody volume='x-soft'>I'm speaking softly.</prosody><prosody volume='x-loud'> And now, This is loud.</prosody>
Customizing pronunciation with the phoneme
tag#
We can use the phoneme
tag to override the pronunciation of words from the predicted pronunciation. For a given word or sequence of words, use the ph
attribute to provide an explicit pronunciation, and the alphabet
attribute to provide the phone set.
Starting with the Riva 2.8.0 release, ipa
will be the only supported prounciation alphabet for TTS models. Older Riva models only support x-arpabet
.
IPA#
For the full list of supported ipa
phonemes, refer to the Riva TTS Phoneme Support page.
Arpabet#
The full list of phonemes in the CMUdict can be found at cmudict.phone. The list of supported symbols with stress can be found at cmudict.symbols. For a mapping of these phones to English sounds, refer to the ARPABET Wikipedia page.
Let’s look at an example showing this custom pronunciation for Riva TTS:
# Setting up Riva TTS request with SynthesizeSpeechRequest
"""
Raw text is "You say tomato, I say tomato."
We are updating this raw text with SSML:
1. Envelope raw text in '<speak>' tags as is required for SSML
2. For a substring in the raw text, add '<phoneme>' tags with 'alphabet' attribute set to 'x-arpabet'
(currently the only supported value) and 'ph' attribute set to a custom pronunciation based on CMUdict and ARPABET
"""
raw_text = "You say tomato, I say tomato."
ssml_text = '<speak>You say <phoneme alphabet="ipa" ph="təˈmeɪˌtoʊ">tomato</phoneme>, I say <phoneme alphabet="ipa" ph="təˈmɑˌtoʊ">tomato</phoneme>.</speak>'
# Older arpabet version
# ssml_text = '<speak>You say <phoneme alphabet="x-arpabet" ph="{@T}{@AH0}{@M}{@EY1}{@T}{@OW2}">tomato</phoneme>, I say <phoneme alphabet="x-arpabet" ph="{@T}{@AH0}{@M}{@AA1}{@T}{@OW2}">tomato</phoneme>.</speak>'
print("Raw Text: ", raw_text)
print("SSML Text: ", ssml_text)
req["text"] = ssml_text
# Request to Riva TTS to synthesize audio
resp = riva_tts.synthesize(**req)
# Playing the generated audio from Riva TTS request
audio_samples = np.frombuffer(resp.audio, dtype=np.int16)
ipd.display(ipd.Audio(audio_samples, rate=sample_rate_hz))
Expected results if you run the tutorial:#
You say <phoneme alphabet="ipa" ph="təˈmeɪˌtoʊ">tomato</phoneme>, I say <phoneme alphabet="ipa" ph="təˈmɑˌtoʊ">tomato</phoneme>.
Here are more examples showing the customization of pronunciation in generated audio:
# SSML texts we want to try
ssml_texts = [
"""<speak>You say <phoneme ph="ˈdeɪtə">data</phoneme>, I say <phoneme ph="ˈdætə">data</phoneme>.</speak>""",
"""<speak>Some people say <phoneme ph="ˈɹut">route</phoneme> and some say <phoneme ph="ˈɹaʊt">route</phoneme>.</speak>""",
]
# Older arpabet version
# ssml_texts = [
# """<speak>You say <phoneme alphabet="x-arpabet" ph="{@D}{@EY1}{@T}{@AH0}">data</phoneme>, I say <phoneme alphabet="x-arpabet" ph="{@D}{@AE1}{@T}{@AH0}">data</phoneme>.</speak>""",
# """<speak>Some people say <phoneme alphabet="x-arpabet" ph="{@R}{@UW1}{@T}">route</phoneme> and some say <phoneme alphabet="x-arpabet" ph="{@R}{@AW1}{@T}">route</phoneme>.</speak>""",
# ]
# Loop through 'ssml_texts' list and synthesize audio with Riva TTS for each entry 'ssml_texts'
for ssml_text in ssml_texts:
req["text"] = ssml_text
resp = riva_tts.synthesize(**req)
audio_samples = np.frombuffer(resp.audio, dtype=np.int16)
print("SSML Text: ", ssml_text)
ipd.display(ipd.Audio(audio_samples, rate=sample_rate_hz))
print("--------------------------------------------")
Expected results if you run the tutorial:#
You say <phoneme ph="ˈdeɪtə">data</phoneme>, I say <phoneme ph="ˈdætə">data</phoneme>.
Some people say <phoneme ph="ˈɹut">route</phoneme> and some say <phoneme ph="ˈɹaʊt">route</phoneme>.
Replacing pronunciation with the sub
tag#
We can use the sub
tag to replace the pronounciation of the specified word or phrase with a different word or phrase. You can specify the pronunciation to substitute with the alias attribute.
# Setting up Riva TTS request with SynthesizeSpeechRequest
"""
Raw text is "WWW is know as the web"
We are updating this raw text with SSML:
1. Envelope raw text in '<speak>' tags as is required for SSML
2. Add '<sub>' tag with 'alias' attribute set to replace www with `World Wide Web`
"""
raw_text = "WWW is know as the web."
ssml_text = '<speak><sub alias="World Wide Web">WWW</sub> is known as the web.</speak>'
print("Raw Text: ", raw_text)
print("SSML Text: ", ssml_text)
req["text"] = ssml_text
# Request to Riva TTS to synthesize audio
resp = riva_tts.synthesize(**req)
# Playing the generated audio from Riva TTS request
audio_samples = np.frombuffer(resp.audio, dtype=np.int16)
ipd.display(ipd.Audio(audio_samples, rate=sample_rate_hz))
Expected results if you run the tutorial:#
<sub alias="World Wide Web">WWW</sub> is known as the web.
Emphasize words with the emphasis
tag#
Use the emphasis tag to emphasize words. Use the riva-build
command with the enable_emphasis_tag
, start_of_emphasis_token
, and end_of_emphasis_token
tags to enable the emphasis feature. The emphasis tag should be used per word basis. If the word ends with a punctuation, only the word will be emphasized and not the punctuation.
Limitation#
The emphasis tag is training data dependent and is available only in the English-US
model. The models which are trained without the emphasis tag in the training data will not result in emphasized speech. Input text containing more than one word wrapped by the emphasis tag is an invalid input. Space wrapped inside the emphasis tag is also an invalid input.
# Setting up Riva TTS request with SynthesizeSpeechRequest
"""
Raw text is "Hello World"
We are updating this raw text with SSML:
1. Envelope raw text in '<speak>' tags as is required for SSML
2. Add '<emphasis>' tag around `love`
"""
ssml_texts = [
"""<speak>I would <emphasis>love</emphasis> to try that.</speak>""",
"""<speak><emphasis>Wow!</emphasis> Thats really cool.</speak>"""
]
print("Raw Text: ", raw_text)
print("SSML Text: ", ssml_text)
for ssml_text in ssml_texts:
req["text"] = ssml_text
resp = riva_tts.synthesize(**req)
# Playing the generated audio from Riva TTS request
audio_samples = np.frombuffer(resp.audio, dtype=np.int16)
print("SSML Text: ", ssml_text)
ipd.display(ipd.Audio(audio_samples, rate=sample_rate_hz))
print("--------------------------------------------")