On-Prem

A100

FastPitch HiFi-GAN en-US

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	22	24.2	25	25.3	2.84	3.1	3.15	4.02	150.8
4	40	50	60	70	5	8	9	12	340
8	63	84	90	100	8	12	14	18	420
16	120	143	154	200	14.3	17.8	19.4	23	460
32	323	340	355	390	14.5	17.9	19.9	23.9	440

Magpie TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	73	76	77	78	20	21	22	22	4.29874
8	20	21	22	22	42	44	46	48	16.9543
16	216	239	251	280	49	63	66	69	36.9589
32	327	364	368	373	67	74	75	95	68.0908
64	618	690	708	743	145	168	170	183	108.443

Magpie TTS Zeroshot

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	373	371	406	440	88	105	108	114	1.036
2	4080	7515	8111	8680	110	160	200	218	1.08

Magpie TTS Flow

# of streams	Throughput (RTFX)
1	3.94927
2	4.22968
4	4.32184
6	4.33257

H100

FastPitch HiFi-GAN en-US

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	17	19	19.3	20	2.5	3.035	3.08	3.16	185
4	30	42	50	60	4	6	7	9	430
8	60	80	80	90	6	10	11	14	500
16	100	120	130	2000	7.7	13	14.6	18.2	500
32	200	230	242	500	9.5	13	14.6	18.63	700

Magpie TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	63	66	67	68	18	19	19	20	4.99748
8	172	181	182	185	35	36	37	39	20.0845
16	184	204	210	223	40	49	51	53	42.7873
32	260	283	295	350	49	54	57	77	82.1573
64	450	485	495	517	99	114	116	124	135.728

Magpie TTS Zeroshot

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	265	265	295	309	59	65	67	71	1.553
2	2740	5525	5672	6118	80	124	135	149	1.529

Magpie TTS Flow

# of streams	Throughput (RTFX)
1	6.03871
2	6.60842
4	6.72419
6	6.74131

L40

FastPitch HiFi-GAN en-US

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	21.5	24.3	24.7	25.5	2.4	3.3	3.5	4	162
4	40	55	60	70	5	7	8	10	300
8	60	80	86	100	6.8	10	11	13	440
16	100	122	133	170	9.7	14.4	16.4	21	600
32	300	310	320	2000	12	17	19.4	24	500

Magpie TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	43	46	47	48	9	10	10	11	9.72692
8	94	99	100	103	18	19	20	21	38.1726
16	116	136	142	157	24	32	34	38	69.5305
32	210	226	230	264	39	44	45	48	105.495

Magpie TTS Zeroshot

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	327	329	344	361	77	93	96	102	1.182
2	3967	6917	7213	8187	94	139	161	190	1.2

Magpie TTS Flow

# of streams	Throughput (RTFX)
1	4.8313
2	5.09014
4	5.22121
6	5.28647