Performance

Evaluation Process#

This section presents latency and throughput numbers of the Riva text-to-speech (TTS) service on different GPUs. Performance of the TTS service was measured for a different number of parallel streams. Each parallel stream performed 20 iterations over 10 input strings from the LJSpeech dataset. Each stream sends a request to the Riva server and waits for all audio chunks to have been received before sending another request. Latency to first audio chunk, latency between successive audio chunks, and throughput were measured. The following diagram shows how the latencies are measured.

Schematic Diagram of Latencies Measured by Riva Streaming TTS Client

The FastPitch and HiFi-GAN models were tested.

The Riva TTS perf client riva_tts_perf_client, provided in the Riva image, was used to measure performance. The source code of the client can be obtained from https://github.com/nvidia-riva/cpp-clients.

The following command was used to generate the tables below:

riva_tts_perf_client \
    --num_parallel_requests=<num_streams> \
    --voice_name=English-US.Female-1 \
    --num_iterations=<20*num_streams> \
    --online=true \
    --text_file=$test_file \
    --write_output_audio=false

Where test_file is a path to the ljs_audio_text_test_filelist_small.txt file.

Results#

Latencies to first audio chunk, latencies between audio chunks, and throughput are reported in the following tables. Throughput is measured in RTFX (duration of audio generated / computation time).

For specifications of the hardware on which these measurements were collected, refer to the Hardware Specifications section.

Note

--num_iterations equals 100 for Xavier AGX, Xavier NX, and Orin AGX and 20 for all other measurements.

Note

The values in the tables are average values over 3 trials. The values in the table are rounded to the last significant digit according to standard deviation calculated on 3 trials. If a standard deviation is less than 0.001 of the average, then the corresponding value is rounded as if standard deviation equals 0.001 of the value.

Please notice, that results on AWS and GCP are computed using Riva 2.4.0 and results on-prem on Riva 2.8.0.

Cloud instance descriptions for AWS and GCP.

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	20	23.2	50	60	3.14	4	4.04	4	140
4	43	60	70	100	5	8	10	13	320
6	60	83	88	100	6	10.1	12	15	360
8	73	97	110	140	7.4	12.2	14.5	18.6	400
10	80	107	116	200	8	13	15	19	420

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	20	24	50	60	4.11	4.98	5.1	5.6	120
4	50	70	80	100	6.7	11	14	19	260
6	80	105	110	140	7.5	14.6	17	20.5	280
8	100	130	140	200	8.8	17	20	25	300
10	114	150	160	200	9.4	18	21	28	320

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	24	24.2	50	60	4	5	5.1	6	120
4	50	80	90	100	7	11.5	14	19	250
6	84	112	118	160	7.6	15	17	21	270
8	103	133	140	200	8.5	16.8	19.4	25	300
10	120	154	164	200	9.03	18	21	28	310

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	20	24	50	60	3.9	4.4	4.96	5.3	120
4	57	80	87	100	7.5	13	16	21	230
6	94	124	130	200	8.6	18	20	24	240
8	118	155	170	200	10	19	22	30	260
10	140	184	190	230	11	21	24	34	264

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	20	21.2	40	50	2.67	3.074	3.14	4	150
4	30	43	50	80	3.92	6.04	7	9	420
6	50	70	70	100	4.6	8	9	12	450
8	60	80	86	100	5.3	9.6	11.26	14.3	510
10	68	90	96	100	5.6	10	12.2	16	530

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	30	28	60	60	5.41	6.1	6.2	6.45	100
4	62	90	100	100	8.6	14.6	17	26	205
6	105	143.4	154	180	9.8	17	20.5	27	213
8	130	180	190	250	12	19	22	28	230
10	150	200	210	300	13	20	23	30	237

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	30	31	60	90	6.95	8.27	9	11	80
4	86	123	135	200	15.3	25.7	30	43	132
6	144	200	220	300	19.2	31.7	36	50	136
8	200	265	280	400	22.7	37	42	55	140
10	230	310	330	400	25	40	46	60	142

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	24	24.3	50	50	3.57	4.1	4.4	5.2	130
4	50	70	80	100	7	13	15.15	20	250
6	80	110	120	140	9.4	18.5	21	25	255
8	115	148	156	200	11	21	24	30	260
10	133	177	190	200	12.4	24	28	36	262

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	20	22.3	40	50	2.54	3.2	3.3	3.7	160
4	40	60	60	100	4	6	7	8	350
6	60	80	80	100	5	7	8	10	400
8	80	100	100	140	5.5	8.5	9	11.5	400
10	90	120	123	150	6	9	10.2	13	430

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	64.4	67.9	68.2	68.9	1.465	1.645	1.842	18.72	21.62

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	97.6	102.6	103.1	103.6	2.84	2.6	3.485	28.4	14.2

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	32.6	32.95	33.1	134.8	4.86	5.72	5.74	6.72	38.25

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	30	31	40	60	6.9	7.2	7.5	9	80
4	94	125	140	200	17.5	30	36	44	118
6	150	200	220	300	25	42	50	63	117
8	190	260	290	350	34.4	55	64	79.7	119
10	216	300	330	400	38	61	70	87	119.2

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	76	77	78	90	44	48	48	50	18
4	690	940	1000	1100	70	115	148	200	22.6
6	1130	1510	1590	1750	91	170	210	296	22.5
8	1637	2100	2200	2400	101	186	230	320	22.53
10	2050	2500	2610	2900	110	200	246	340	22.9

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	30	27	40	40	5.05	6.2	7	8	100
4	53	70	80	100	9.4	16.1	18.2	24	215
6	77	108.4	120	150	12.25	20.8	23.8	30	233
8	100	140	150	180	15.5	25.6	30	40	240
10	113	157	170	200	17.2	28	32	42	245

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	60	54.8	55	56	27.4	29.6	30.2	31	28
4	330	444	470	530	30.8	47	56	77	48.9
6	500	650	690	770	35	57	70	100	53.4
8	670	860	900	990	39	68	83	120	55.9
10	840	1050	1140	1350	43	76	94	130	56.5

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	30	30	50	60	5.6	6.17	6.24	6.4	90
4	57	80	90	100	9.3	16	17.5	26	210
6	82	120	130	150	12	19.2	22	28	230
8	113	158	170	200	15	24	27	33	230
10	122	170	184	200	17	26.8	30	37	236

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	66	67.6	68.4	70	33.1	35.8	36	37	23
4	356	480	510	564	32	50	57	76	45.5
6	530	700	740	800	37	60	70	100	50.4
8	710	930	980	1060	40	66	80	110	53.2
10	870	1070	1160	1360	43	73	90	130	55

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	30	25.1	40	50	3.8	4.7	5.1	5.4	120
4	38	50	60	90	5.9	9.97	11.7	15	320
6	51	70	76	100	7.2	12.4	14.6	19	370
8	63	85	90	100	8.5	15	17	23	400
10	68	94	102	120	9.3	17	20	26	420

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	40	42.3	42.7	45	19.9	21.8	22.4	23	38
4	194	268	285	310	18	27	30	36	83
6	270	350	370	410	20	30	36	47	97
8	350	450	470	510	21	34	40	50	105
10	430	525	560	680	22.4	35	41	56	109

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	30	26	40	50	3.4	4.04	4.14	4.6	120
4	40	55	60	90	5.3	9	10	13	320
6	54	75	80	100	6.7	11.1	13.02	17	370
8	64	87	95	100	8.2	14.07	16.4	21.5	410
10	68	94	100	120	9	16	19	25	430

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	45	45.1	45.8	50	21.33	24	24.1	24.6	36
4	197	280	296	320	18	27	31	37	83
6	267	345	360	394	19	30	34	42	99
8	336	430	450	500	21.6	34	39	50	108
10	414	496	526	660	22.3	35	40	53	113

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	30	34.4	50	70	7.7	8.2	8.4	10	70
4	98	135	150	200	18.4	31	36	46	113
6	156	225	240	300	25	40.5	45	57	114
8	205	290	310	360	33	52	60	72	116
10	230	320	340	400	37	56	63	80	118

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	80	83	84	100	44.6	48.6	49.2	50.4	17.6
4	650	900	950	1050	69	116	144	190	23.4
6	1056	1390	1470	1600	85	157	190	270	23.97
8	1520	2000	2100	2270	95	174	220	300	24.25
10	1890	2350	2500	3000	105	200	240	350	24.45

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	30	27.6	40	50	5.6	6.16	6.24	6.5	100
4	58	80.5	90	100	9.8	16	18.5	28	200
6	83	120	130	150	13	20.6	24	32	220
8	112.3	157	170	200	16	26	30	37	223
10	124	172	183	200	18.3	28.4	32	40	227

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	63	65	66	67	32.5	35.1	35.7	36.4	24
4	366	500	520	600	32.7	50	60	80	44.7
6	540	710	754	820	37	60	75	106	49.5
8	720	930	980	1080	40.5	68	83	115	52.7
10	900	1100	1200	1400	43.4	76	95	130	53.5

On-Prem Hardware Specifications#

GPU
NVIDIA DGX A100 40 GB
CPU
Model	AMD EPYC 7742 64-Core Processor
Thread(s) per core	2
Socket(s)	2
Core(s) per socket	64
NUMA node(s)	8
Frequency boost	enabled
CPU max MHz	2250
CPU min MHz	1500
RAM
Model	Micron DDR4 36ASF8G72PZ-3G2B2 3200MHz
Configured Memory Speed	2933 MT/s
RAM Size	32x64GB (2048GB Total)

GPU
NVIDIA A30
CPU
Model	AMD EPYC 7742 64-Core Processor
Thread(s) per core	1
Socket(s)	2
Core(s) per socket	64
NUMA node(s)	2
Frequency boost	disabled
CPU max MHz	2250.0000
CPU min MHz	1500.0000
RAM
Model	Samsung DDR4 M393A4K40DB3-CWE 3200MHz
Configured Memory Speed	3200 MT/s
RAM Size	32x64GB (2048GB Total)

GPU
NVIDIA A10
CPU
Model	AMD EPYC 7763 64-Core Processor
Thread(s) per core	1
Socket(s)	2
Core(s) per socket	64
NUMA node(s)	8
Frequency boost	enabled
CPU max MHz	2450
CPU min MHz	1500
RAM
Model	Samsung DDR4 M393A4K40DB3-CWE 3200 MHz
Configured Memory Speed	3200 MT/s
RAM Size	16x32GB (512GB Total)

GPU
NVIDIA V100 SXM2 16 GB
CPU
Model	Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
Thread(s) per core	2
Socket(s)	2
Core(s) per socket	20
NUMA node(s)	2
CPU max MHz	3600
CPU min MHz	1200
RAM
Model	Micron DDR4 36ASF4G72PZ-2G6D1 2667MHz
Configured Memory Speed	2133 MT/s
RAM Size	16x32GB (512GB Total)

GPU
NVIDIA T4
CPU
Model	Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz
Thread(s) per core	2
Socket(s)	2
Core(s) per socket	18
NUMA node(s)	2
CPU max MHz	3900
CPU min MHz	1000
RAM
Model	Samsung DDR4 M393A2K43BB1-CTD 2666MHz
Configured Memory Speed	2666 MT/s
RAM Size	24x16GB (384GB Total)

Performance Considerations#

When the server is under high load, requests might time out, as the server will not start inference for a new request until a previous request is completely generated so that inference slot can be freed. This is done to maximize throughput for the TTS service and allow for real-time interaction.

NVIDIA Riva

Contents

Performance#

Evaluation Process#

Results#

On-Prem Hardware Specifications#

Performance Considerations#