On-Prem

A100

Magpie TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	79.37	92.15	95.65	411.16	12.16	13.66	13.93	15.14	12.2
2	100.99	114.86	121.08	121.08	13.96	15.04	24.76	29.21	20.94
4	116.53	160.98	173.41	181.82	15.25	17.74	27.24	40.1	37.4
8	199.94	275.91	392.05	555.17	24.5	35.27	43.0	69.18	55.4
16	239.29	317.42	416.23	443.88	38.26	60.34	70.82	95.46	86.79
32	211.8	256.56	268.2	285.62	56.02	72.96	83.97	120.37	149.62
64	395.17	478.86	498.11	515.6	123.18	164.86	207.76	284.59	197.31

Magpie TTS Zeroshot

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	142.21	147.05	148.33	152.49	12.17	13.88	14.15	14.72	10.52
2	176.70	228.20	256.48	271.31	14.35	14.84	15.36	57.43	14.52
4	213.24	278.84	294.94	355.52	19.92	19.46	60.66	101.21	22.58
8	315.69	429.04	479.42	606.65	31.35	69.17	101.37	116.61	31.00
16	1584.10	1977.05	2059.59	2209.28	39.47	93.30	114.97	195.98	33.73

Magpie TTS Flow

# of streams	Throughput (RTFX)
1	3.94927
2	4.22968
4	4.32184
6	4.33257

Chatterbox TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	527.47	532.18	533.35	549.47	485.18	521.37	523.98	527.16	1.43
2	594.78	683.04	695.75	695.75	491.99	539.70	607.11	664.94	2.67
4	700.22	819.65	837.37	890.56	514.43	633.89	678.19	722.55	5.00
8	784.56	970.46	1166.24	1465.38	547.36	692.56	757.88	860.11	8.51
16	926.78	1162.61	1339.91	2165.71	597.32	837.57	1046.47	1845.82	16.25
32	2268.96	3693.08	4635.20	6645.20	849.18	1156.59	2265.26	3032.59	20.55
64	3746.55	5616.85	6452.63	8958.53	1302.96	1851.52	2297.68	3971.97	23.90

H100

Magpie TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	46.78	52.96	55.1	57.05	10.16	13.36	14.07	15.1	14.69
2	58.38	64.79	65.42	65.42	10.48	11.04	11.6	21.6	27.82
4	61.83	71.21	71.75	72.01	11.78	12.41	13.34	23.6	51.22
8	74.6	89.13	91.98	97.87	15.67	19.65	20.76	34.87	89.81
16	92.31	108.79	127.2	129.41	21.93	25.5	27.33	45.73	151.16
32	147.67	174.32	182.78	188.62	38.85	48.32	62.76	86.08	220.12
64	275.35	328.69	346.54	356.53	89.88	121.52	153.73	212.69	290.79

Magpie TTS Zeroshot

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	179.36	196.66	199.52	202.53	11.64	15.99	18.14	24.56	12.00
2	195.91	236.69	246.82	267.82	13.31	17.77	20.54	40.56	17.78
4	226.49	285.79	304.72	344.22	18.01	27.19	42.25	68.96	30.91
8	288.45	388.15	422.94	504.87	29.38	66.03	73.11	97.70	48.43
16	421.06	596.43	657.92	789.28	54.80	100.54	106.97	132.09	66.25
32	686.98	1020.34	1154.41	1492.62	103.18	151.95	171.98	193.06	78.53

Magpie TTS Flow

# of streams	Throughput (RTFX)
1	6.03871
2	6.60842
4	6.72419
6	6.74131

Chatterbox TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	423.94	429.91	438.80	445.88	369.21	400.24	402.70	409.05	1.85
2	474.99	543.06	549.29	549.29	376.28	415.45	468.22	494.15	3.52
4	536.60	658.44	752.02	777.40	390.85	506.86	533.22	615.36	6.55
8	588.99	754.41	780.87	838.02	417.08	543.64	563.76	698.93	11.19
16	835.23	1175.01	1741.68	3234.17	468.67	721.56	942.42	1367.18	20.63
32	1391.40	1767.97	3442.42	3826.35	542.81	727.59	832.81	2368.11	32.87
64	2746.89	4092.95	6124.62	7842.92	968.19	1347.06	1948.58	4272.14	32.93

L40

Magpie TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	42.8	46.93	47.15	47.96	8.6	9.12	9.27	10.3	18.34
2	65.08	70.12	70.65	70.65	13.91	15.14	15.56	28.34	25.92
4	55.19	63.37	68.23	68.45	10.72	11.95	12.51	22.71	61.42
8	66.88	73.91	82.43	82.93	14.12	15.91	16.39	28.05	102.83
16	96.47	111.37	115.77	131.04	23.47	26.71	28.3	48.89	139.47
32	170.08	195.2	198.39	203.92	45.36	55.91	68.51	99.52	189.02

Magpie TTS Zeroshot

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	108.46	119.48	124.70	124.70	12.45	14.26	14.80	17.54	12.03
2	123.52	157.26	166.27	174.73	12.83	13.64	15.20	36.81	20.58
4	151.04	197.14	205.38	361.94	16.25	21.61	35.42	58.16	32.44
8	195.98	274.14	331.49	501.76	23.66	53.63	58.63	61.65	49.01
16	918.21	1155.15	1213.08	1615.71	29.39	59.84	64.21	138.55	59.06
32	2732.86	3035.23	3109.38	3949.16	30.52	60.76	70.64	151.79	59.30

Magpie TTS Flow

# of streams	Throughput (RTFX)
1	4.8313
2	5.09014
4	5.22121
6	5.28647

Chatterbox TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	394.96	396.26	396.54	398.57	336.20	363.12	364.26	367.14	2.02
2	447.74	551.80	573.31	573.31	351.59	386.18	419.66	455.27	3.79
4	496.33	582.65	592.32	659.66	367.37	449.65	476.30	529.05	6.99
8	523.10	640.65	678.32	890.31	373.07	478.13	540.94	596.96	13.74
16	863.12	1555.76	1745.35	1931.19	403.35	696.65	851.44	1263.60	22.64
32	1321.85	1858.44	1901.75	2427.36	504.82	752.50	861.44	1069.70	33.70
64	2311.47	4061.38	4281.49	5069.28	796.91	1123.11	1297.64	2804.21	40.44

B200

Magpie TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	32.39	35.23	35.9	36.64	7.55	5.44	6.13	6.62	12.13
2	36.18	41.26	41.39	41.39	8.01	5.44	5.98	7.27	22.95
4	41.19	49.22	49.5	51.0	11.21	8.21	8.69	10.89	42.06
8	51.07	65.27	71.25	74.06	16.39	13.02	14.08	19.51	79.07
16	71.44	91.52	114.8	122.59	29.52	24.14	28.5	637.49	135.96
32	117.86	172.26	178.29	198.13	62.34	57.66	70.16	1683.36	215.62
64	239.13	314.21	358.84	445.44	158.9	160.02	216.8	2608.34	319.81

Chatterbox TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	352.94	355.54	356.76	411.93	286.32	308.77	310.50	313.72	2.36
2	384.05	449.78	488.24	488.24	294.96	324.26	344.36	371.69	4.50
4	438.47	509.67	536.94	567.73	323.41	382.54	407.61	489.88	8.02
8	456.86	537.34	539.37	546.89	323.56	409.28	423.95	469.68	15.95
16	588.68	785.38	981.76	1225.87	365.26	488.98	560.08	816.19	27.21
32	1139.84	1528.95	1714.38	2109.88	451.16	627.62	679.19	1367.68	36.69
64	2239.96	4264.38	5004.28	7587.56	775.94	1059.72	1309.91	3254.42	41.70

DGX_Spark

Magpie TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	53.03	56.65	57.26	59.25	8.71	9.78	10.12	12.31	9.81
2	65.54	72.39	79.19	79.19	10.23	12.27	14.67	18.91	17.14
8	125.04	148.43	150.63	161.48	26.78	37.33	40.85	56.54	40.29
16	214.09	251.27	270.51	283.51	52.68	72.58	81.1	110.9	53.68
32	432.48	533.05	558.14	626.81	126.82	178.24	197.99	279.69	63.26
64	961.72	1242.8	1299.5	1375.72	329.78	496.19	569.41	786.63	75.88

Magpie TTS Zeroshot

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	238.1	252.0	253.1	253.1	8.5	10.7	12.3	13.7	8.5
2	306.6	384.4	406.6	418.1	12.8	16.2	17.6	65.4	13.0
4	377.8	556.0	567.9	744.5	26.1	37.9	85.2	130.9	18.9
8	546.6	831.5	905.7	1398.4	56.7	109.2	164.9	179.7	24.1
16	893.6	1217.7	1406.3	2782.7	128.0	215.8	253.1	273.8	27.9
32	1743.8	2340.1	2703.8	4823.2	304.4	408.5	424.5	486.2	30.3

Chatterbox TTS Multilingual

# of streams	Latency to first audio (ms)				Latency between audio chunks (ms)				Throughput (RTFX)
	avg	p90	p95	p99	avg	p90	p95	p99
1	357.52	427.04	437.72	459.94	240.72	282.86	316.49	351.08	2.74
2	436.87	495.28	606.57	606.57	294.94	350.69	368.72	431.68	4.47
4	616.65	792.50	807.07	894.74	368.98	491.29	522.38	619.14	6.75
8	885.82	1082.45	1150.24	1389.14	477.11	660.66	711.70	871.46	10.17
16	1717.39	2572.25	3080.30	4594.59	769.74	1181.94	1472.03	2152.18	12.07
32	2813.28	3753.18	3990.96	4873.21	1241.74	1816.23	2046.19	2322.51	15.02
64	6670.24	9271.26	11739.70	14130.90	2409.61	3749.86	4152.97	5401.64	14.50