Is this page helpful?

Customizing TTS Models#

The options on this page customize synthesis at request time. To include persistent customizations in a model, such as a fixed pronunciation dictionary, custom text-normalization rules, or a custom voice, build and deploy a custom NIM as described in Deploying Custom Models as NIM.

SSML Customization#

Speech Synthesis Markup Language (SSML) controls the performance of the virtual speaker. The TTS NIM microservice supports a subset of SSML that lets you override pronunciation for specific words.

The following SSML tags are supported:

<phoneme>: Overrides pronunciation for specific words.
<say-as>: Steers text normalization for a token using a specified interpretation class.

SSML Support by Model#

Model	Phoneme	Say-As	Custom Dictionary
Magpie TTS Multilingual	✅	✅ (en-US, hi-IN, ja-JP)	✅
Magpie TTS Zeroshot	✅	✅ (en-US)	✅
Chatterbox TTS Multilingual	❌	❌	❌

Note

All SSML inputs must be a valid XML document wrapped in a <speak> root tag. Input that is not valid XML, or valid XML with a different root tag, is treated as raw text.

Example#

Customize pronunciation with the phoneme tag:

python3 python-clients/scripts/tts/talk.py --server 0.0.0.0:50051 \
    --text "<speak>You say <phoneme alphabet='ipa' ph='təˈmeɪˌtoʊ'>tomato</phoneme>, I say <phoneme alphabet='ipa' ph='təˈmɑˌtoʊ'>tomato</phoneme>.</speak>" \
    --language-code en-US

The synthesized audio file output.wav contains the resulting speech with the phoneme overrides applied.

Say-As#

The <say-as interpret-as="..."> tag tells the normalizer how to read a token. It is supported for English (en-US), Hindi (hi-IN), and Japanese (ja-JP) on Magpie TTS Multilingual, and for English (en-US) on Magpie TTS Zeroshot.

Supported interpret-as values:

Value	Behavior	Example input	Example output
`cardinal`	Reads a number as a cardinal integer	`42`	“forty two”
`ordinal`	Reads a number as an ordinal	`12`	“twelfth”
`digits`	Reads each digit individually (numeric input only)	`911`	“nine one one”
`characters`	Reads each character individually	`ABC`	“a b c”
`telephone`	Reads a phone number	`555-1234`	“five five five one two three four”
`time`	Reads a time expression	`3:30pm`	“three thirty pm”
`date`	Reads a date expression	`01/05/2025`	“January fifth twenty twenty five”
`fraction`	Reads a fraction	`1/2`	“one half”

Example#

python3 python-clients/scripts/tts/talk.py --server 0.0.0.0:50051 \
    --text "<speak>Call us at <say-as interpret-as='telephone'>555-1234</say-as>.</speak>" \
    --language-code en-US

Custom Pronunciation Dictionary#

The TTS NIM microservice supports custom pronunciation through a text-based dictionary that maps words (graphemes) to IPA phonetic representations (phonemes). Use the --custom-dictionary flag to pass the dictionary file to the client.

Dictionary format:

Each line contains a word followed by its pronunciation, separated by exactly two spaces.
Split multi-word entries into individual lines.
Refer to Phoneme Support for the list of supported IPA phonemes.

Example#

python3 python-clients/scripts/tts/talk.py --server 0.0.0.0:50051 \
    --text "Today is a sunny day, a great day to eat fresh tomato" \
    --language-code en-US \
    --custom-dictionary custom_dict.txt

The custom dictionary file custom_dict.txt contains word-to-phoneme mappings:

sunny  ˈsʌnɪ
tomato  ˈtɑˌməʊ

The synthesized audio file output.wav contains the resulting speech with the custom pronunciations applied.

Emotion Exaggeration#

Chatterbox TTS Multilingual accepts a per-request exaggeration_factor parameter that controls how pronounced the emotional prosody is. Pass it through the --custom-configuration flag, which forwards comma-separated key:value pairs to the underlying model.

Parameter	Type	Range	Default
`exaggeration_factor`	float	`0.25` – `2.0`	`0.5`

Model Support#

Model	Emotion Exaggeration
Chatterbox TTS Multilingual	✅
Magpie TTS Multilingual	❌
Magpie TTS Zeroshot	❌

Note

Values outside [0.25, 2.0] are rejected by the server with INVALID_ARGUMENT (gRPC) or HTTP 400 (Bad Request, invalid custom_configuration).

Example#

python3 python-clients/scripts/tts/talk.py --server 0.0.0.0:50051 \
    --text "I cannot believe this just happened!" \
    --language-code en-US \
    --voice Chatterbox-Multilingual.en-US.Male \
    --custom-configuration "exaggeration_factor:1.5"

To pass multiple parameters at once, separate key:value pairs with commas:

... --custom-configuration "exaggeration_factor:1.5,key2:value2"

Streaming Text Flush Controls#

When using the streaming gRPC endpoint (SynthesizeOnline) or the WebSocket realtime API, you can control how buffered plain text is flushed to synthesis using custom_configuration keys.

Key	Type	Default	Description
`flush`	`"true"` or `"1"`	unset	Flushes the current text buffer immediately without waiting for a punctuation boundary.
`chunk_len_threshold`	integer string	model `max_sequence_length`	Minimum buffered character count after which a punctuation boundary can trigger a flush.
`max_chunk_threshold`	integer string	model `max_sequence_length`	Maximum buffered characters before forcing a flush regardless of punctuation.

A value of 0 for chunk_len_threshold or max_chunk_threshold falls back to the model’s max_sequence_length.

Model Support#

Model	Flush Controls
Magpie TTS Multilingual	✅
Magpie TTS Zeroshot	✅
Chatterbox TTS Multilingual	❌

Word Timestamps#

Set enable_word_time_offsets to true in SynthesizeSpeechRequest to receive per-word timing information alongside the synthesized audio. The response meta.words field contains a list of word entries, each with word, start_time, and end_time in milliseconds.

Model Support#

Model	Word Timestamps
Magpie TTS Multilingual	✅
Magpie TTS Zeroshot	✅
Chatterbox TTS Multilingual	❌

Example#

request = riva.proto.riva_tts_pb2.SynthesizeSpeechRequest(
    text="Hello world.",
    language_code="en-US",
    encoding=riva.proto.riva_audio_pb2.LINEAR_PCM,
    sample_rate_hz=22050,
    voice_name="Magpie-Multilingual.EN-US.Aria",
    enable_word_time_offsets=True,
)
response = tts_client.Synthesize(request)
for word_info in response.meta.words:
    print(f"{word_info.word}: {word_info.start_time}ms – {word_info.end_time}ms")

The word field reflects the normalized text after preprocessing, not necessarily the raw input token, and word boundaries come from splitting on whitespace, so original spacing isn’t preserved. For SynthesizeOnline, word timestamps are computed per segment internally but delivered to the client as a single consolidated timeline on the final (often audio-less) response of the stream, not incrementally per chunk. Use the flush custom configuration key to force an earlier segment boundary, which shortens how much text has to accumulate before its timestamps are included.