<no title>

On-Prem

H100

Parakeet-0.6B-CTC

English (US)

Streaming, low-latency

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 270

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	12.7	13.4	13.7	40	0.999
8	15.3	14.3	15.5	20	45.5	7.99
16	21	18	26	28	60	15.97
32	28	28	37	40	90	31.9
48	35	35	46	47.4	100	47.8
64	42	40	54	55	130	63.7

Streaming, high-throughput

Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1240

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	13.7	20	30	100	1
64	53	50	64	120	150	63.7
128	77	65	96	200	245	127
256	120	107	156	300	440	252.5
384	162	145	220	440	640	376
512	200	180	276	530	800	499

Offline

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	300
False	32	2200
True	32	170

Parakeet-1.1B-CTC

English (US)

Streaming, low-latency

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	16.7	17.4	20	40	0.999
8	21	19.8	21	30	50.4	7.99
16	29	26	40	41	70	15.96
32	41	45	51	55	110	31.9
48	53	58	71	73	160	47.75
64	65.5	72	83	85	210	63.6

Streaming, high-throughput

Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	18.4	20	36	100	0.999
64	80	83	95	140	200	63.7
128	119	100	155	230	296	126.8
256	190	170	270	400	530	252
384	260	245	378	580	800	374.5
512	346	330	496	850	1200	494

Offline

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	200
False	32	2000
True	32	120

Parakeet-1.1B-RNNT

Multilingual

Streaming, low-latency

Chunk size (ms): 320
Language model: n-gram
Maximum effective # of streams with n-gram language model: 5

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	100	95	103.5	105	200	0.994
8	7600	7000	16000	17000	19000	5.25

Streaming, high-throughput

Chunk size (ms): 1600
Language model: n-gram
Maximum effective # of streams with n-gram language model: 25

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	100	98	111	112	300	0.997
64	27000	24000	50000	50000	54000	25

Offline

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	80
False	32	130

Conformer-CTC

Spanish (US)

Streaming, low-latency

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 355

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10	9.9	11.3	12	40	1
8	12.6	12	13.4	17	31	8
16	17	15	22	25	40	15.98
32	23	23	31	33	50	31.94
48	29	28	40	41	70	47.9
64	33.6	38	45	47	70	63.9
128	49	47	64	67	150	127.6
256	84	75	107	126	391	255

Streaming, high-throughput

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1400

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	11	20	40	80	1
64	39	40	55	80	110	63.9
128	58	50	75	150	202	127.6
256	90	80	115	240	380	255
384	120	107	155	316	530	381.4
512	149	130	196	400	700	508
768	258	200	630	680	1280	756
1024	420	263	1280	1350	1900	992

Offline

Language model: n-gram

# of streams	Throughput (RTFX)
32	467

Whisper-Large

Multilingual

Offline

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	90
False	32	370

Canary-1B

Multilingual

Offline

Language model: none

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	11
False	32	77

Canary-0.6B-Turbo

Multilingual

Offline

Language model: none

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	40
False	32	300

A100

Parakeet-CTC-0.6B

English (US)

Streaming, low-latency

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 179

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	15.3	16.2	16.3	40	0.999
8	21.6	20.4	22	23	59	7.99
16	28	26.4	30	39	80	15.96
32	41.4	40	53	54	130	31.85
48	49	54	64	66	160	47.7
64	59	67	75	76	216	63.6

Streaming, high-throughput

Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 810

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	19.6	30	40	100	0.999
64	90	93	110	200	240	63.5
128	115	100	140	260	350	126.6
256	185	163	248	451	630	251
384	254	230	350	630	930	373
512	362	300	730	940	1550	491

Offline

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	300
False	32	2000
True	32	125

Parakeet-CTC-1.1B

English (US)

Streaming, low-latency

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 104

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24	22.7	23.7	25	50	0.999
8	32.7	31	33	51	72.7	7.98
16	44	40.8	50	63	110	15.94
32	59	60	73	75	180	31.8
48	79	90	93	100	240	47.6
64	100	109	114	160	310	63.4

Streaming, high-throughput

Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 490

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	29.1	40	50	100	0.999
64	123	130	160	240	260	63.5
128	185	165	240	360	430	126.4
256	300	266	430	630	830	249.4
384	460	445	770	1100	1560	368
512	720	650	1400	1550	2150	483

Offline

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	180
False	32	1330
True	32	75

Conformer-CTC

Spanish (US)

Streaming, low-latency

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 233

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	11.8	12.8	14	40	1
8	17.6	16.8	18.5	22	39	8
16	22.5	21.3	25	31	60.3	15.98
32	32.4	35	42	46	70	31.93
48	41	40	58	59	100	47.9
64	46	50	64	66	100	63.8
128	73	66	94	97	220	127.5

Streaming, high-throughput

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 980

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	13	20	40	80	1
64	60	60	80	110	180	63.8
128	90	80	110	230	300	127.5
256	133.3	120	174	340	530	254
384	183	166	245	430	800	380
512	260	223	510	600	1200	505
768	535	354	1500	1640	2150	739
1024	940	600	2300	2570	2930	960

Offline

Language model: n-gram

# of streams	Throughput (RTFX)
32	460

Whisper-Large

Multilingual

Offline

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	60
False	32	234

Canary-1B

Multilingual

Offline

Language model: none

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	5.7
False	32	38.75

Canary-0.6B-Turbo

Multilingual

Offline

Language model: none

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	24
False	32	168

L40

Parakeet-CTC-0.6B

English (US)

Streaming, low-latency

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 190

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19	18.3	19.3	20	43.5	0.999
8	24	23	30	30	65	7.98
16	31.4	29	38.3	42	80	15.96
32	42	42	57	60	100	31.9
48	52	53	69.6	75	130	47.8

Streaming, high-throughput

Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 900

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25	22	30	50	90	0.999
64	90	90	110	160	200	63.6
128	120	100	150	240	330	126.8
256	180	160	240	400	560	251.5

Offline

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	240
False	32	2030
True	32	101.5

Parakeet-CTC-1.1B

English (US)

Streaming, low-latency

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 110

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25	23	29	30	50	0.999
8	31	29	35.5	46	70	7.98
16	44	40	56	60	100	15.95
32	60	62	76	80	150	31.84
48	80	86	100	112	227	47.7

Streaming, high-throughput

Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 578

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	27	40	50	100	0.999
64	120	130	150	200	240	63.5
128	170	150	220	310	380	126.5
256	270	250	390	540	700	250.5

Offline

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	180
False	32	1440
True	32	94

Conformer-CTC

Spanish (US)

Streaming, low-latency

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 280

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	10.3	11.2	12.4	30	1
8	20	19	26	30	42	7.99
16	28	26	35	40	56	15.97
32	35	35	48	52	73	31.9
64	50	55	66	70	100	63.8

Streaming, high-throughput

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1180

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	11.5	20	30	60	1
64	70	70	90	100	170	63.8
128	88	84	110	190	250	127.4
256	128	117	164	300	460	254.4

Offline

Language model: n-gram

# of streams	Throughput (RTFX)
32	440

Whisper-Large

Multilingual

Offline

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	70
False	32	193.5

Canary-1B

Multilingual

Offline

Language model: none

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	6.2
False	32	43.3

Canary-0.6B-Turbo

Multilingual

Offline

Language model: none

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	28
False	32	192