On-Prem

A100

FastPitch HiFi-GAN en-US

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	22	24.2	25	25.3	2.84	3.1	3.15	4.02	150.8
4	40	50	60	70	5	8	9	12	340
8	63	84	90	100	8	12	14	18	420
16	120	143	154	200	14.3	17.8	19.4	23	460
32	323	340	355	390	14.5	17.9	19.9	23.9	440

Magpie TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	327	304	371	1333	95	109	112	115	1.43
2	3457	6387	6625	7917	107	129	132	137	1.56

Magpie TTS Zeroshot

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	373	371	406	440	88	105	108	114	1.036
2	4080	7515	8111	8680	110	160	200	218	1.08

H100

FastPitch HiFi-GAN en-US

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	17	19	19.3	20	2.5	3.035	3.08	3.16	185
4	30	42	50	60	4	6	7	9	430
8	60	80	80	90	6	10	11	14	500
16	100	120	130	2000	7.7	13	14.6	18.2	500
32	200	230	242	500	9.5	13	14.6	18.63	700

Magpie TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	209	210	234	238	68	78	79	82	2
2	689	2178	2439	2712	96	113	116	123	2.5

Magpie TTS Zeroshot

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	265	265	295	309	59	65	67	71	1.553
2	2740	5525	5672	6118	80	124	135	149	1.529

L40

FastPitch HiFi-GAN en-US

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	21.5	24.3	24.7	25.5	2.4	3.3	3.5	4	162
4	40	55	60	70	5	7	8	10	300
8	60	80	86	100	6.8	10	11	13	440
16	100	122	133	170	9.7	14.4	16.4	21	600
32	300	310	320	2000	12	17	19.4	24	500

Magpie TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	253	261	274	283	76	87	89	92	1.78
2	1707	4593	5099	5573	97	118	123	128	2.08

Magpie TTS Zeroshot

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	327	329	344	361	77	93	96	102	1.182
2	3967	6917	7213	8187	94	139	161	190	1.2