On-Prem

A100

Magpie TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	77.83	83.76	85.61	125.94	12.72	14.82	15.42	19.24	11.9
2	212.53	278.13	279.33	279.33	19.04	26.0	40.8	60.18	14.37
4	221.46	389.04	397.59	429.57	25.01	42.19	52.59	77.34	25.37
8	175.62	358.76	376.96	380.77	24.02	32.54	37.95	53.29	59.66
16	181.36	339.59	357.65	371.05	44.21	69.04	86.1	105.89	94.04
32	596.39	809.9	878.25	995.5	181.98	388.49	468.24	555.08	113.14
64	435.66	569.43	601.21	701.97	174.37	249.57	259.91	386.02	204.31

Magpie TTS Zeroshot

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	142.21	147.05	148.33	152.49	12.17	13.88	14.15	14.72	10.52
2	176.70	228.20	256.48	271.31	14.35	14.84	15.36	57.43	14.52
4	213.24	278.84	294.94	355.52	19.92	19.46	60.66	101.21	22.58
8	315.69	429.04	479.42	606.65	31.35	69.17	101.37	116.61	31.00
16	1584.10	1977.05	2059.59	2209.28	39.47	93.30	114.97	195.98	33.73

Chatterbox TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
8	516.96	1480.01	1481.48	1486.95	219.35	308.02	330.29	384.55	21.06
32	952.14	1309.66	1324.28	1441.49	418.79	478.32	499.04	798.53	43.49
64	1457.13	1786.69	1955.94	1977.99	743.36	833.11	865.91	963.75	51.65

H100

Magpie TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	35.12	41.57	43.55	45.83	10.93	14.72	15.13	15.94	14.92
2	45.24	53.16	59.37	59.37	10.38	11.34	11.58	12.03	29.94
4	63.88	69.85	153.58	153.64	13.74	16.63	18.04	20.38	45.88
8	76.76	96.92	188.71	190.41	17.94	23.68	25.19	30.39	85.68
16	79.49	96.27	100.48	110.67	26.15	32.8	36.31	41.93	150.63
32	162.03	215.15	292.64	771.55	58.48	81.23	95.06	206.06	214.28
64	266.82	331.71	425.82	567.64	115.69	168.34	188.93	220.25	308.54

Magpie TTS Zeroshot

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	179.36	196.66	199.52	202.53	11.64	15.99	18.14	24.56	12.00
2	195.91	236.69	246.82	267.82	13.31	17.77	20.54	40.56	17.78
4	226.49	285.79	304.72	344.22	18.01	27.19	42.25	68.96	30.91
8	288.45	388.15	422.94	504.87	29.38	66.03	73.11	97.70	48.43
16	421.06	596.43	657.92	789.28	54.80	100.54	106.97	132.09	66.25
32	686.98	1020.34	1154.41	1492.62	103.18	151.95	171.98	193.06	78.53

Chatterbox TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
8	446.75	1404.85	1406.47	1409.63	155.39	198.60	207.19	241.52	28.20
32	666.15	1084.71	1098.77	1304.23	294.63	340.41	354.85	367.45	62.87
64	1050.95	1351.16	1717.77	1737.06	504.83	604.37	636.19	675.57	75.59

L40S

Magpie TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	32.16	37.52	38.24	40.59	9.49	10.28	10.44	10.76	17.28
2	43.91	49.97	50.46	50.46	10.15	11.13	11.5	14.19	33.46
4	46.89	54.0	58.78	62.76	11.31	12.56	12.78	13.51	62.09
8	54.32	64.88	65.56	66.76	17.12	19.11	19.27	19.8	97.93
16	79.92	94.65	97.89	99.78	26.13	32.11	33.25	36.31	156.08
32	133.03	156.9	159.59	161.97	49.0	61.06	64.1	72.01	203.26

Magpie TTS Zeroshot

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	108.46	119.48	124.70	124.70	12.45	14.26	14.80	17.54	12.03
2	123.52	157.26	166.27	174.73	12.83	13.64	15.20	36.81	20.58
4	151.04	197.14	205.38	361.94	16.25	21.61	35.42	58.16	32.44
8	195.98	274.14	331.49	501.76	23.66	53.63	58.63	61.65	49.01
16	918.21	1155.15	1213.08	1615.71	29.39	59.84	64.21	138.55	59.06
32	2732.86	3035.23	3109.38	3949.16	30.52	60.76	70.64	151.79	59.30

Chatterbox TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
8	453.50	1377.92	1381.35	1382.31	184.69	249.13	264.22	289.65	24.95
32	701.43	1144.00	1155.95	1250.00	318.65	365.71	375.42	396.76	58.80
64	1208.51	1448.26	1673.11	1695.22	586.00	682.57	717.89	766.93	65.37

B200

Magpie TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	32.39	35.23	35.9	36.64	7.55	5.44	6.13	6.62	12.13
2	36.18	41.26	41.39	41.39	8.01	5.44	5.98	7.27	22.95
4	41.19	49.22	49.5	51.0	11.21	8.21	8.69	10.89	42.06
8	51.07	65.27	71.25	74.06	16.39	13.02	14.08	19.51	79.07
16	71.44	91.52	114.8	122.59	29.52	24.14	28.5	637.49	135.96
32	117.86	172.26	178.29	198.13	62.34	57.66	70.16	1683.36	215.62
64	239.13	314.21	358.84	445.44	158.9	160.02	216.8	2608.34	319.81

Chatterbox TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
32	1242.32	8102.87	8122.34	8140.74	229.20	280.13	290.65	317.54	59.95

DGX_Spark

Magpie TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	53.03	56.65	57.26	59.25	8.71	9.78	10.12	12.31	9.81
2	65.54	72.39	79.19	79.19	10.23	12.27	14.67	18.91	17.14
8	125.04	148.43	150.63	161.48	26.78	37.33	40.85	56.54	40.29
16	214.09	251.27	270.51	283.51	52.68	72.58	81.1	110.9	53.68
32	432.48	533.05	558.14	626.81	126.82	178.24	197.99	279.69	63.26
64	961.72	1242.8	1299.5	1375.72	329.78	496.19	569.41	786.63	75.88

Magpie TTS Zeroshot

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	238.1	252.0	253.1	253.1	8.5	10.7	12.3	13.7	8.5
2	306.6	384.4	406.6	418.1	12.8	16.2	17.6	65.4	13.0
4	377.8	556.0	567.9	744.5	26.1	37.9	85.2	130.9	18.9
8	546.6	831.5	905.7	1398.4	56.7	109.2	164.9	179.7	24.1
16	893.6	1217.7	1406.3	2782.7	128.0	215.8	253.1	273.8	27.9
32	1743.8	2340.1	2703.8	4823.2	304.4	408.5	424.5	486.2	30.3

Chatterbox TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
8	663.48	1313.35	1321.71	1328.73	486.30	609.73	639.45	686.24	10.63