{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "# NVIDIA TTS NIM Tutorial\n", "\n", "This tutorial walks you through the various features of NVIDIA TTS NIM and how to use the APIs in a Python application. NVIDIA TTS NIM uses the gRPC API to serve offline and online use cases." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Prerequisites\n", "\n", "1. Deploy NVIDIA TTS NIM with the Magpie Multilingual model using the [NVIDIA TTS NIM documentation](https://docs.nvidia.com/nim/speech/latest/tts/index.html).\n", "2. Install the Riva Python Client library.\n", "\n", " ```bash\n", " sudo apt-get install python3-pip\n", " pip install -U nvidia-riva-client\n", " ```\n", "3. Clone the Git repository at https://github.com/nvidia-riva/tutorials for audio samples. The repository is assumed to be cloned in the `$HOME` directory.\n", "\n", " ```bash\n", " cd $HOME\n", " git clone https://github.com/nvidia-riva/tutorials.git\n", " ```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Import Riva Client Libraries\n", "\n", "Import the necessary libraries, including the Riva Client libraries." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# Import required libraries\n", "import io\n", "import json\n", "import wave\n", "from pathlib import Path\n", "import riva.client\n", "import IPython.display as ipd" ], "execution_count": 1, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Create utility functions." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# save synthesized audio to wav file on disk\n", "def save_audio_to_file(filename, audio, output_sample_rate):\n", " with wave.open(filename, \"wb\") as wav_file:\n", " wav_file.setnchannels(1)\n", " wav_file.setsampwidth(2)\n", " wav_file.setframerate(output_sample_rate)\n", " wav_file.writeframes(audio)\n", "\n", "# list available voices\n", "def list_voices(tts_service):\n", " request = riva.client.proto.riva_tts_pb2.RivaSynthesisConfigRequest()\n", " response = tts_service.stub.GetRivaSynthesisConfig(request)\n", "\n", " tts_models = dict()\n", " for model_config in response.model_config:\n", " language_code = model_config.parameters['language_code']\n", " voice_name = model_config.parameters['voice_name']\n", " subvoices = [voice.split(':')[0] for voice in model_config.parameters['subvoices'].split(',')]\n", " full_voice_names = [voice_name + \".\" + subvoice for subvoice in subvoices]\n", " tts_models[language_code] = full_voice_names\n", "\n", " print(json.dumps(tts_models, indent = 4))" ], "execution_count": 2, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Inference Modes\n", "\n", "NVIDIA TTS NIM supports both streaming and offline inference modes. In offline mode, response audio is returned only after the full audio sequence for the requested text is generated. In streaming or online mode, response audio is received in chunks as it is generated. This significantly reduces the latency for large requests, particularly the time to first audio." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following sections demonstrate available models in NVIDIA TTS NIM." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Synthesize using Magpie TTS Multilingual Model\n", "\n", "This section assumes that you have deployed the **Magpie TTS Multilingual** model. Refer to the [NVIDIA TTS NIM tutorial](https://docs.nvidia.com/nim/speech/latest/get-started/tutorials/tts.html) for deployment instructions.\n", "\n", "The **Magpie TTS Multilingual** model supports both offline and online inference modes." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Create a Riva client and query the supported languages and voices." ] }, { "cell_type": "code", "metadata": {}, "source": [ "auth = riva.client.Auth(uri='0.0.0.0:50051')\n", "tts_service = riva.client.SpeechSynthesisService(auth)\n", "list_voices(tts_service)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Perform offline inference using `Synthesize` API. Response is received when the entire audio is synthesized. Output is saved to file `output.wav`." ] }, { "cell_type": "code", "metadata": { "scrolled": false }, "source": [ "output_sample_rate = 44100\n", "request = riva.client.proto.riva_tts_pb2.SynthesizeSpeechRequest(\n", " text = \"Experience the future of speech AI with Riva, where every word comes to life with clarity and emotion.\",\n", " language_code = \"en-US\",\n", " encoding = riva.client.AudioEncoding.LINEAR_PCM,\n", " sample_rate_hz = output_sample_rate,\n", " voice_name = \"Magpie-Multilingual.EN-US.Aria\" # Change according to available voices\n", ")\n", "\n", "response = tts_service.stub.Synthesize(request)\n", "\n", "save_audio_to_file(\"output.wav\", response.audio, output_sample_rate)\n", "ipd.Audio(\"output.wav\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Perform online/streaming inference using `SynthesizeOnline` API. Responses are received as soon as the audio chunks are synthesized." ] }, { "cell_type": "code", "metadata": {}, "source": [ "responses = tts_service.stub.SynthesizeOnline(request)\n", "bytes_buffer = io.BytesIO()\n", "for response in responses:\n", " bytes_buffer.write(response.audio)\n", "\n", "save_audio_to_file(\"output_online.wav\", bytes_buffer.getvalue(), output_sample_rate)\n", "ipd.Audio(\"output_online.wav\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Refer to the [NVIDIA TTS NIM API Reference](https://docs.nvidia.com/nim/speech/latest/reference/api-references/tts/protos.html) for more details about API usage." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Synthesize using Magpie TTS Zeroshot Model\n", "\n", "This section assumes that you have deployed the **Magpie TTS Zeroshot** model. Refer to the [NVIDIA TTS NIM tutorial](https://docs.nvidia.com/nim/speech/latest/get-started/tutorials/tts.html) for deployment instructions.\n", "\n", "The **Magpie TTS Zeroshot** model supports text to speech using an input text and audio prompt. Voice characteristics from the audio prompt are applied to the synthesized output speech. The following sections demonstrate the model capability using the sample Python client.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Create a Riva client and query the supported languages and voices." ] }, { "cell_type": "code", "metadata": {}, "source": [ "auth = riva.client.Auth(uri='0.0.0.0:50051')\n", "tts_service = riva.client.SpeechSynthesisService(auth)\n", "list_voices(tts_service)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Load audio prompt to be used to synthesize speech." ] }, { "cell_type": "code", "metadata": {}, "source": [ "zero_shot_audio_prompt_file = Path(\"~/tutorials/audio_samples/tts_samples/sample_audio_prompt.wav\").expanduser() # Path to the audio prompt file\n", "with zero_shot_audio_prompt_file.open('rb') as f:\n", " audio_data = f.read()\n", " audio_prompt_data = audio_data\n", "if audio_prompt_data is None:\n", " raise ValueError(\"Audio prompt data is empty. Please check the file path and content.\")\n", "\n", "zero_shot_data = riva.client.proto.riva_tts_pb2.ZeroShotData(\n", " audio_prompt = audio_prompt_data,\n", " quality = 32,\n", ")" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Perform online/streaming inference using `SynthesizeOnline` API. Responses are received as soon as the audio chunks are synthesized." ] }, { "cell_type": "code", "metadata": {}, "source": [ "output_sample_rate = 44100\n", "request = riva.client.proto.riva_tts_pb2.SynthesizeSpeechRequest(\n", " language_code = \"en-US\",\n", " encoding = riva.client.AudioEncoding.LINEAR_PCM,\n", " sample_rate_hz = output_sample_rate,\n", " text = \"Experience the future of speech AI with Riva, where every word comes to life with clarity and emotion.\",\n", " voice_name = \"Magpie-ZeroShot-Multilingual\",\n", " zero_shot_data = zero_shot_data,\n", ")\n", "\n", "responses = tts_service.stub.SynthesizeOnline(request)\n", "bytes_buffer = io.BytesIO()\n", "for response in responses:\n", " bytes_buffer.write(response.audio)\n", "\n", "save_audio_to_file(\"output_magpie_zero_shot_online.wav\", bytes_buffer.getvalue(), output_sample_rate)\n", "ipd.Audio(\"output_magpie_zero_shot_online.wav\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Refer to the [NVIDIA TTS NIM API Reference](https://docs.nvidia.com/nim/speech/latest/reference/api-references/tts/protos.html) for more details about API usage." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Synthesize using Magpie TTS Flow Model\n", "\n", "This section assumes that you have deployed the **Magpie TTS Flow** model. Refer to the [NVIDIA TTS NIM tutorial](https://docs.nvidia.com/nim/speech/latest/get-started/tutorials/tts.html) for deployment instructions.\n", "\n", "The **Magpie TTS Flow** model supports text to speech using an input text and audio prompt. Voice characteristics from the audio prompt are applied to the synthesized output speech. The following sections demonstrate the model capability using the sample Python client.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Create a Riva client and query the supported languages and voices." ] }, { "cell_type": "code", "metadata": {}, "source": [ "auth = riva.client.Auth(uri='0.0.0.0:50051')\n", "tts_service = riva.client.SpeechSynthesisService(auth)\n", "list_voices(tts_service)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Magpie TTS Flow model supports only offline API." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following code loads the audio prompt to be used for speech synthesis and performs offline inference using `Synthesize` API. The response is received when the entire audio is synthesized." ] }, { "cell_type": "code", "metadata": {}, "source": [ "zero_shot_audio_prompt_file = Path(\"~/tutorials/audio_samples/tts_samples/sample_audio_prompt.wav\").expanduser() # Path to the audio prompt file\n", "with zero_shot_audio_prompt_file.open('rb') as f:\n", " audio_data = f.read()\n", " audio_prompt_data = audio_data\n", "if audio_prompt_data is None:\n", " raise ValueError(\"Audio prompt data is empty. Please check the file path and content.\")\n", "\n", "zero_shot_data = riva.client.proto.riva_tts_pb2.ZeroShotData(\n", " audio_prompt = audio_prompt_data,\n", " quality = 32,\n", " transcript = \"I consent to use my voice to create a synthetic voice.\"\n", ")\n", "\n", "request = riva.client.proto.riva_tts_pb2.SynthesizeSpeechRequest(\n", " language_code = \"en-US\",\n", " encoding = riva.client.AudioEncoding.LINEAR_PCM,\n", " sample_rate_hz = output_sample_rate,\n", " text = \"Experience the future of speech AI with Riva, where every word comes to life with clarity and emotion.\",\n", " zero_shot_data = zero_shot_data,\n", ")\n", "response = tts_service.stub.Synthesize(request)\n", "save_audio_to_file(\"output_magpie_flow.wav\", response.audio, output_sample_rate)\n", "ipd.Audio(\"output_magpie_flow.wav\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Customizing Riva TTS audio output with SSML\n", "\n", "Speech Synthesis Markup Language (SSML) specification is a markup for directing the performance of the virtual speaker. Riva supports portions of SSML, allowing you to adjust pitch, rate, and pronunciation of the generated audio.\n", "\n", "All SSML inputs must be a valid XML document and use the root tag. All non-valid XML and all valid XML with a different root tag are treated as raw input text.\n", "\n", "Riva TTS supports the following SSML tags:\n", "\n", "- The ``phoneme`` tag, which allows us to control the pronunciation of the generated audio.\n", " \n", "Let's look at customization of Riva TTS with these SSML tags in some detail." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Customizing pronunciation with the `phoneme` tag\n", "\n", "We can use the `phoneme` tag to override the pronunciation of words from the predicted pronunciation. For a given word or sequence of words, use the `ph` attribute to provide an explicit pronunciation, and the `alphabet` attribute to provide the phone set.\n", "\n", "Riva TTS supports `ipa` as the only supported pronunciation alphabet for TTS models. For the full list of supported `ipa` phonemes, refer to the [Riva TTS Phoneme Support](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tts/tts-phones.html) page." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Examples showing the customization of pronunciation in generated audio using Phoneme tag." ] }, { "cell_type": "code", "metadata": { "scrolled": true }, "source": [ "# SSML text examples with Phoneme tag\n", "\"\"\"\n", "Instructions for using Phoneme tag:\n", "1. Envelope raw text in '' tags as is required for SSML\n", "2. For a substring in the raw text, add '' tags with 'alphabet' attribute set to 'ipa'\n", " (currently the only supported value) and 'ph' attribute set to a custom IPA pronunciation\n", "\"\"\"\n", "ssml_texts = [\n", " \"\"\"You say tomato, I say tomato.\"\"\",\n", "]\n", "\n", "# Loop through 'ssml_texts' list and synthesize audio with Riva TTS for each entry 'ssml_texts'\n", "for i, ssml_text in enumerate(ssml_texts):\n", " request.text = ssml_text\n", " response = tts_service.stub.Synthesize(request)\n", " save_audio_to_file(f\"output_ssml_{i}.wav\", response.audio, 44100)\n", " print(f\"Synthesized audio for SSML Text: {ssml_text}\")\n", " ipd.display(ipd.Audio(f\"output_ssml_{i}.wav\", rate=44100))\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Customizing Riva TTS audio output with custom pronunciation dictionary\n", "\n", "Riva TTS supports providing a text dictionary to get the desired pronunciation for specific words synthesized by the server. This custom dictionary must contain a word (grapheme) followed by the desired pronunciation (phoneme), both separated by two spaces. Different such words and pronunciation pairs can be provided on a new line in the input dictionary file. The input dictionary file can be passed in the custom_dictionary field while configuring a request from the client.\n", "\n", "Example showing the customization of pronunciation in generated audio using custom dictionary:" ] }, { "cell_type": "code", "metadata": {}, "source": [ "# Custom dictionary text examples\n", "\"\"\"\n", "Instructions for using custom_dictionary:\n", "1. Use raw text for the request text.\n", "2. Set request.custom_dictionary to pronunciation entries in the format: word, two spaces, IPA pronunciation.\n", "\"\"\"\n", "texts = [\n", " \"You say tomato, I say tomato.\",\n", "]\n", "custom_dictionary = \"tomato təˈmeɪˌtoʊ\"\n", "\n", "# Loop through 'texts' list and synthesize audio with Riva TTS for each entry.\n", "for i, text in enumerate(texts):\n", " request.text = text\n", " request.custom_dictionary = custom_dictionary\n", " response = tts_service.stub.Synthesize(request)\n", " save_audio_to_file(f\"output_custom_dictionary_{i}.wav\", response.audio, 44100)\n", " print(f\"Synthesized audio for text: {text}\")\n", " ipd.display(ipd.Audio(f\"output_custom_dictionary_{i}.wav\", rate=44100))" ], "execution_count": null, "outputs": [] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 2 }