On-Prem

H100

Parakeet-CTC-1.1B

English (US)

Streaming, low-latency

Chunk size (ms): 160

Language model: n-gram

Maximum effective # of streams with n-gram language model: 104

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	16.7	17.4	20	40	0.999
8	21	19.8	21	30	50.4	7.99
16	29	26	40	41	70	15.96
32	41	45	51	55	110	31.9
48	53	58	71	73	160	47.75
64	65.5	72	83	85	210	63.6

Streaming, high-throughput

Chunk size (ms): 960

Language model: n-gram

Maximum effective # of streams with n-gram language model: 490

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	18.4	20	36	100	0.999
64	80	83	95	140	200	63.7
128	119	100	155	230	296	126.8
256	190	170	270	400	530	252
384	260	245	378	580	800	374.5
512	346	330	496	850	1200	494

Offline

Language model: n-gram

# of streams	Throughput (RTFX)
1	200
32	2000

Conformer-CTC

Spanish (US)

Streaming, low-latency

Chunk size (ms): 160

Language model: n-gram

Maximum effective # of streams with n-gram language model: 355

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10	9.9	11.3	12	40	1
8	12.6	12	13.4	17	31	8
16	17	15	22	25	40	15.98
32	23	23	31	33	50	31.94
48	29	28	40	41	70	47.9
64	33.6	38	45	47	70	63.9
128	49	47	64	67	150	127.6
256	84	75	107	126	391	255

Streaming, high-throughput

Chunk size (ms): 800

Language model: n-gram

Maximum effective # of streams with n-gram language model: 1400

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	11	20	40	80	1
64	39	40	55	80	110	63.9
128	58	50	75	150	202	127.6
256	90	80	115	240	380	255
384	120	107	155	316	530	381.4
512	149	130	196	400	700	508
768	258	200	630	680	1280	756
1024	420	263	1280	1350	1900	992

Offline

Language model: n-gram

# of streams	Throughput (RTFX)
32	467

A100

Parakeet-CTC-1.1B

English (US)

Streaming, low-latency

Chunk size (ms): 160

Language model: n-gram

Maximum effective # of streams with n-gram language model: 104

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24	22.7	23.7	25	50	0.999
8	32.7	31	33	51	72.7	7.98
16	44	40.8	50	63	110	15.94
32	59	60	73	75	180	31.8
48	79	90	93	100	240	47.6
64	100	109	114	160	310	63.4

Streaming, high-throughput

Chunk size (ms): 960

Language model: n-gram

Maximum effective # of streams with n-gram language model: 490

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	29.1	40	50	100	0.999
64	123	130	160	240	260	63.5
128	185	165	240	360	430	126.4
256	300	266	430	630	830	249.4
384	460	445	770	1100	1560	368
512	720	650	1400	1550	2150	483

Offline

Language model: n-gram

# of streams	Throughput (RTFX)
1	180
32	1330

Conformer-CTC

Spanish (US)

Streaming, low-latency

Chunk size (ms): 160

Language model: n-gram

Maximum effective # of streams with n-gram language model: 233

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	11.8	12.8	14	40	1
8	17.6	16.8	18.5	22	39	8
16	22.5	21.3	25	31	60.3	15.98
32	32.4	35	42	46	70	31.93
48	41	40	58	59	100	47.9
64	46	50	64	66	100	63.8
128	73	66	94	97	220	127.5
256	770	700	1300	1540	1800	242.6

Streaming, high-throughput

Chunk size (ms): 800

Language model: n-gram

Maximum effective # of streams with n-gram language model: 980

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	13	20	40	80	1
64	60	60	80	110	180	63.8
128	90	80	110	230	300	127.5
256	133.3	120	174	340	530	254
384	183	166	245	430	800	380
512	260	223	510	600	1200	505
768	535	354	1500	1640	2150	739
1024	940	600	2300	2570	2930	960

Offline

Language model: n-gram

# of streams	Throughput (RTFX)
32	460

L40

Parakeet-CTC-1.1B

English (US)

Streaming, low-latency

Chunk size (ms): 160

Language model: n-gram

Maximum effective # of streams with n-gram language model: 104

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25	23	29	30	50	0.999
8	31	29	35.5	46	70	7.98
16	44	40	56	60	100	15.95
32	60	62	76	80	150	31.84
48	80	86	100	112	227	47.7

Streaming, high-throughput

Chunk size (ms): 960

Language model: n-gram

Maximum effective # of streams with n-gram language model: 490

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	27	40	50	100	0.999
64	120	130	150	200	240	63.5
128	170	150	220	310	380	126.5
256	270	250	390	540	700	250.5

Offline

Language model: n-gram

# of streams	Throughput (RTFX)
1	180
32	1440

Conformer-CTC

Spanish (US)

Streaming, low-latency

Chunk size (ms): 160

Language model: n-gram

Maximum effective # of streams with n-gram language model: 280

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	10.3	11.2	12.4	30	1
8	20	19	26	30	42	7.99
16	28	26	35	40	56	15.97
32	35	35	48	52	73	31.9
64	50	55	66	70	100	63.8

Streaming, high-throughput

Chunk size (ms): 800

Language model: n-gram

Maximum effective # of streams with n-gram language model: 1180

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	11.5	20	30	60	1
64	70	70	90	100	170	63.8
128	88	84	110	190	250	127.4
256	128	117	164	300	460	254.4

Offline

Language model: n-gram

# of streams	Throughput (RTFX)
32	440