Performance#

Evaluation Process#

This section shows the latency and throughput numbers for streaming and offline configurations of the Riva ASR service on different GPUs. These numbers were captured after the pre-configured ASR pipelines from our Quick Start scripts were deployed. The Conformer and Parakeet acoustic models were tested.

In streaming mode, the client and the server used audio chunks of the same duration (100ms, 160ms, and 800ms depending on the server configuration). Refer to the Results section for the chunk size value to use.

The Riva streaming client riva_streaming_asr_client, provided in the Riva image, was used with the --simulate_realtime flag to simulate transcription from a microphone, where each stream was doing three iterations over a sample audio file (1272-135031-0000.wav) from the LibriSpeech dev-clean dataset. The LibriSpeech datasets can be obtained from https://www.openslr.org/12.

The source code for the riva_streaming_asr_client can be obtained from https://github.com/nvidia-riva/cpp-clients.

The command used to measure performance was:

riva_streaming_asr_client \
   --chunk_duration_ms=<chunk_duration> \
   --simulate_realtime=true \
   --automatic_punctuation=true \
   --num_parallel_requests=<num_streams> \
   --word_time_offsets=false \
   --print_transcripts=false \
   --interim_results=false \
   --num_iterations=<3*num_streams> \
   --audio_file=1272-135031-0000.wav \
   --output_filename=/tmp/output.json

The riva_streaming_asr_client returns the following latency measurements:

intermediate latency: latency of responses returned with is_final == false
final latency: latency of responses returned with is_final == true
latency: the overall latency of all returned responses. This is what is tabulated in the following tables.

Refer to the following diagram for a schematic representation of the different latencies measured by the Riva streaming ASR client.

Schematic Diagram of Latencies Measured by Riva Streaming ASR Client

In offline mode, the command used to measure maximum throughput was:

riva_asr_client \
   --automatic_punctuation=true \
   --num_parallel_requests=32 \
   --word_time_offsets=false \
   --print_transcripts=false \
   --num_iterations=96 \
   --audio_file=1272-135031-0000x5.wav \
   --output_filename=/tmp/output.json

where 1272-135031-0000x5.wav is simply the 1272-135031-0000.wav audio file concatenated five times. The source code for the riva_asr_client can be obtained from: https://github.com/nvidia-riva/cpp-clients

Results#

Latencies and throughput measurements for streaming and offline configurations are reported in the following tables. Throughput is measured in RTFX (duration of audio transcribed / computation time).

Note

Audio files were iterated 1 time for Xavier AGX, Xavier NX, and Orin AGX and 3 times for all other experiments.

Note

If the language model is none, the inference is performed with a greedy decoder. If the language model is n-gram, then a beam decoder was used.

Note

The values in the tables are average values over 3 trials. The values in the table are rounded to the last significant digit according to standard deviation calculated on 3 trials. If a standard deviation is less than 0.001 of the average, then the corresponding value is rounded as if standard deviation equals 0.001 of the value.

For specifications of the hardware on which these measurements were collected, refer to the Hardware Specifications section. Please notice, that

results on AWS and GCP are computed using Riva 2.4.0
results On-Prem are computed using Riva 2.15.0.

Cloud instance descriptions for AWS and GCP.

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 218
Maximum effective # of streams without language model (greedy generation): 223

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	13	11.9	12.8	13	40	0.999
n-gram	8	18.8	17.4	19	20	57	7.99
n-gram	16	24.8	22	30	32	80	15.96
n-gram	32	34	30	43	46	110	31.86
n-gram	48	44	41	60	66	160	47.7
n-gram	64	50	50	67	75	200	63.6
n-gram	128	86	67	100	220	360	126.5
n-gram	256	1700	1600	3200	3200	3700	229
none	1	12	11.3	12	12.5	30	1
none	8	17	15.8	16.6	20	49.6	7.99
none	16	22.1	19.9	26	29.5	70	15.96
none	32	32	30	39.7	44	100	31.9
none	48	40	40	56	57	160	47.7
none	64	46	45	60	65	170	63.6
none	128	80	60	97	200	330	126.5
none	256	1400	1350	2600	2700	3000	233

Chunk size (ms): 800
Maximum effective # of streams with n-gram language model: 960
Maximum effective # of streams without language model (greedy generation): 970

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	16	13.5	15	31	80	1
n-gram	64	65	69	80	160	210	63.6
n-gram	128	90	80	120	240	330	126.7
n-gram	256	140	110	180	390	600	251.3
n-gram	384	188	165	246	460	850	374
n-gram	512	250	210	550	600	1200	494
n-gram	768	420	310	800	1560	2000	730
n-gram	1024	800	500	2100	2900	3000	953
none	1	14	11.5	12	25	70	1
none	64	58	60	70	120	192	63.6
none	128	80	70	110	230	320	126.8
none	256	127	110	164	300	550	251.5
none	384	175	156	234	443	800	374
none	512	240	200	530	580	1140	495
none	768	410	300	800	1520	1900	731
none	1024	735	480	1970	2730	2900	955

Speaker Diarization	Language model	# of streams	Throughput (RTFX)
False	n-gram	32	2100
False	none	32	2200
True	n-gram	32	84
True	none	32	135

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 219

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	12	14	20	90	0.998
8	19	17.6	20	36.4	38	7.98
16	24.4	22.4	30	42	60	15.96
32	34	35	43	50	93	31.87
48	42	41	58	60	126	47.8
64	48	56	64	67	150	63.6
128	81	70	97	190	300	127
256	710	660	1160	1350	1780	230.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 760

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	14	30	100	100	0.998
64	73	60	140	200	200	63
128	103	80	220	310	320	125
256	168	127	394	500	580	245.4
384	234	190	540	720	880	361
512	340	250	670	970	1320	472
768	650	410	1600	1620	2320	683
1024	1500	1300	2930	3300	3630	826

Language model: n-gram

# of streams	Throughput (RTFX)
32	226

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 233

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	11.8	12.8	14	40	1
8	17.6	16.8	18.5	22	39	8
16	22.5	21.3	25	31	60.3	15.98
32	32.4	35	42	46	70	31.93
48	41	40	58	59	100	47.9
64	46	50	64	66	100	63.8
128	73	66	94	97	220	127.5
256	770	700	1300	1540	1800	242.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 980

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	13	20	40	80	1
64	60	60	80	110	180	63.8
128	90	80	110	230	300	127.5
256	133.3	120	174	340	530	254
384	183	166	245	430	800	380
512	260	223	510	600	1200	505
768	535	354	1500	1640	2150	739
1024	940	600	2300	2570	2930	960

Language model: n-gram

# of streams	Throughput (RTFX)
32	460

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 208

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	11.6	12.4	20	80	0.997
8	18.4	16.5	18	30	60	7.96
16	24.2	21.2	29	31	83	15.9
32	33	30	41	46	115	31.63
48	42	40	57.4	60	170	47.3
64	48	50	63	66	185	62.9
128	83	65	97	240	380	124.2
256	630	600	1080	1200	1600	221

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 660

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	13	50	80	80	0.997
64	80	60	200	200	210	62.8
128	107	80	250	306	320	124
256	172.2	120	396	500	570	242
384	239	188	580	723	860	354.5
512	390	274	880	1170	1420	458
768	930	530	2070	2100	2800	647
1024	2040	2300	4000	4000	4360	716

Language model: n-gram

# of streams	Throughput (RTFX)
32	168

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 227

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	12.8	14	15	40	1
8	18	17.8	19	20	37	7.99
16	23.4	22.7	27	31	42	15.98
32	33.8	37	44	47	52	31.9
48	43	40	60	60	70	47.85
64	46	55	64.6	66.6	76	63.8
128	72	68	100	100	110	127.4
256	950	920	1650	1730	1880	239

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1010

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	14	20	30	100	1
64	60	70	80	90	90	63.8
128	80	80	110	120	140	127.4
256	124	120	177	190	250	253.4
384	171	170	251	300	406	378
512	217	210	326	420	554	501
768	316	307	490	640	860	742
1024	466	445	840	980	1170	977

Language model: n-gram

# of streams	Throughput (RTFX)
32	432

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 231

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	11.8	12.5	12.8	50	1
8	17.6	16.8	18	19	61	7.99
16	23	21.4	26	30	80	15.96
32	33	30	43	45	90	31.9
48	41.4	41.5	58	59	120	47.8
64	46	45	63	65	140	63.6
128	75	64	95	98	280	126.6
256	700	700	1200	1300	1400	242

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 927

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	13	14	27	80	1
64	60	70	80	140	200	63.6
128	90	70	100	230	340	126.6
256	140	113	174	410	630	250.7
384	204	166	370	600	1000	372
512	277	210	750	800	1400	491
768	530	320	1400	2200	2600	720
1024	940	580	2440	3200	3400	940

Language model: n-gram

# of streams	Throughput (RTFX)
32	1600

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 177

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	16.3	18	30	100	0.997
8	27	25	30	33	96	7.95
16	35	35	40	43	120	15.87
32	51	53	60	66	170	31.65
48	57	55	70	90	250	47.3
64	68	70	83	110	300	63
128	130	110	150	400	600	124
256	2700	2580	5200	5600	7000	185

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 560

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	25.5	50	60	100	0.997
64	100	90	300	300	400	63
128	150	127	300	400	500	124.5
256	220	180	500	600	800	244
384	340	260	700	1000	1300	357
512	600	400	1500	1600	2000	465
768	2000	1000	4000	4000	5000	640
1024	4000	5000	7300	8000	8000	676

Language model: n-gram

# of streams	Throughput (RTFX)
32	500

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 233

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12	12.8	13	30	1
8	18	20	20	22	44	7.99
16	23	22	30	31	50	15.97
32	34	37	45	47	60	31.9
48	44	50	59.6	60.6	70	47.8
64	47	56	64.6	66	80	63.8
128	69.5	66	94.4	96	130	127.3
256	600	600	1000	1060	1200	244.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1070

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	13.3	20	30	80	1
64	60	66	80	80	97	63.8
128	77	76	110	118	142	127.3
256	115	110	167	170	203	254
384	152	156	234	240	270	380
512	189	192	300	308	330	505
768	270	270	440	454	560	750
1024	370	370	590	650	860	989

Language model: n-gram

# of streams	Throughput (RTFX)
32	446

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 190

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	13	15.6	30	100	0.997
8	24	20	26	27	100	7.95
16	30	27	34	37	130	15.87
32	40	38	50	50	150	31.66
48	50	45	60	90	200	47.2
64	56	60	70	80	240	63.1
128	100	85	100	300	500	124.7
256	1600	1600	3000	3000	4000	202

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 620

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	19	50	100	100	0.997
64	100	80	200	300	300	63
128	130	100	270	400	500	124.6
256	200	160	400	600	800	244.4
384	280	240	600	780	1000	358
512	400	300	900	1200	1600	467
768	1000	700	3000	3000	3400	662
1024	2800	3000	5000	5600	5800	720

Language model: n-gram

# of streams	Throughput (RTFX)
32	300

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 230

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12	12.6	13	50	0.999
8	18.1	17.3	18.4	20	60	7.98
16	23.5	22	25	30	90	15.95
32	33.4	37	42	46	90	31.86
48	42	40	57	58	160	47.7
64	47	50	64	66	150	63.6
128	75	65	95	98	270	126.5
256	560	550	950	980	1300	243

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 940

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	13	20	50	80	0.999
64	64	60	80	190	197	63.5
128	90	80	100	230	330	126.5
256	140	120	170	380	580	250.5
384	190	163	246	530	860	372
512	261	220	580	700	1260	491.5
768	445	313	1170	1560	2100	723
1024	850	540	2100	2850	3170	941

Language model: n-gram

# of streams	Throughput (RTFX)
32	450

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 190

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	12.2	20	30	100	0.996
8	20	17	40	41	44	7.96
16	27	22	50	60	65	15.85
32	37	34	60	80	100	31.6
48	43	40	75	90	130	47.3
64	51	56	84	100	150	63
128	110	80	230	240	380	123.2
256	810	730	1550	1780	1840	202

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 580

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	13.5	100	100	100	0.995
64	100	70	200	230	230	62
128	134	90	295	300	330	122
256	215	160	480	560	566	235.6
384	304	224	700	800	870	341
512	470	440	940	1130	1340	439
768	1120	1070	1870	2100	2500	600
1024	2250	2700	3800	3800	4400	643

Language model: n-gram

# of streams	Throughput (RTFX)
32	110.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 166

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	13	40	40	100	0.993
8	22	18	54	63	65	7.9
16	30	24	70	80	90	15.7
32	40	34	50	107	120	31.1
48	47	40	80	145	170	46.4
64	60	60	100	160	190	61.2
128	124	77	320	370	470	118.4
256	800	760	1600	1696	2070	178.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 420

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	40	40	100	100	100	0.993
64	120	80	200	200	200	61.3
128	170	100	318	340	350	119
256	277	188	570	580	590	225
384	390	300	780	830	900	320
512	634	600	1100	1300	1380	406
768	1760	1870	2650	2850	3060	462
1024	2800	3100	4300	4500	4800	486

Language model: n-gram

# of streams	Throughput (RTFX)
32	72

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 215

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	12	14	20	80	0.997
8	19	17	20	30	60	7.96
16	25	22	29.5	32	83	15.9
32	35	34	45	50	120	31.65
48	43	42	58	60	150	47.3
64	50	50	65	70	179	63
128	83	68	98	240	350	124.5
256	530	510	900	1040	1300	228

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 700

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	13.3	50	80	80	0.997
64	80	68	200	200	230	62.8
128	110	80	250	300	330	124.3
256	173	125	400	510	570	243
384	243.4	190	566	740	840	356
512	360	270	740	1060	1300	464
768	750	460	1700	1700	2350	665
1024	1800	2000	3600	3600	3950	756

Language model: n-gram

# of streams	Throughput (RTFX)
32	180

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 222

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	11.7	12.7	14	40	1
8	17.7	16.5	18	27	50	7.99
16	23	21	25	30.8	70	15.96
32	33	34	42	47	100	31.86
48	41	40	57	60	160	47.7
64	48	53	64	68	170	63.6
128	83	65	103	210	350	126.5
256	1380	1330	2500	2600	2950	232.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 830

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	12.3	20	40	80	1
64	70	60	170	170	200	63.6
128	110	74	240	300	330	126.6
256	175	140	410	500	570	251
384	245	190	600	740	834	373.5
512	390	290	800	1070	1300	494
768	860	760	1600	1840	2300	728
1024	3000	3000	5000	5000	5400	870

Language model: n-gram

# of streams	Throughput (RTFX)
32	1600

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 210

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	12	13	20	80	0.997
8	19.5	17.5	20	30	67.3	7.96
16	25	22.5	30	34	90	15.9
32	35.2	36	44	45	120	31.67
48	44	42	60	62.5	170	47.35
64	51	50	66	70	200	63

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 759

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	15	50	80	80	0.997
64	80	60	196	200	220	62.9
128	110	80	260	300	360	124.7
256	173	130	420	500	570	244
384	239	190	580	710	830	358.4
512	350	267	750	1000	1300	468

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	193

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 192

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	12.4	17	30	100	0.997
8	21	17.3	36	48	64	7.96
16	28	24	50	70	80	15.9
32	37	32	66	90	113	31.7
48	45	39	80	117	140	47.4
64	54	55	95	146	200	63.1

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 750

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	16	40	100	100	0.997
64	80	70	200	240	240	62.9
128	115	84	260	300	350	124.7
256	175	126	400	500	560	244
384	240	195	570	725	830	359
512	355	270	740	1000	1300	468

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	193

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 13

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	49	47.9	49	50	92	0.997
4	55	52	59	60	106	3.986
8	81.4	95.1	97.6	98.5	170.5	7.95

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 65

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	44	41.3	43	60	100	0.998
4	70	67	80	100	105	3.99
8	90	98	100	120	150	7.96

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	100
False	4	191
False	8	196

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 230

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	13.2	14	14.6	40	1
8	18.4	18	19.4	20.6	36.5	7.99
16	24	23.5	30	32	40	15.97
32	34	30	44	44.5	51	31.9
48	43	42	60	61	73	47.9
64	46	50	64	66	73	63.8
128	72	67	98	100	110	127.4
256	660	660	1100	1150	1240	244

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 980

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	14	20	30	100	1
64	60	70	80	90	90	63.8
128	84	84	120	126	140	127.3
256	127	120	180	200	260	253.3
384	175	170	256	313	420	377.5
512	223	220	335	440	570	500
768	330	320	510	670	890	741
1024	490	470	870	1010	1180	976

Language model: n-gram

# of streams	Throughput (RTFX)
32	203

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 205

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14.5	15.6	16	40	1
8	23	22.6	25	27	40	7.99
16	29.96	29.4	33	37	47	15.97
32	40	42	51	53.2	60	31.9
48	47	47	63	64.7	70	47.9
64	54	60	72	75	90	63.8
128	90	90	120	120	140	127.3
256	3000	2800	5600	6000	7300	210

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 930

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	15.2	20	30	100	1
64	67	80	90	100	100	63.8
128	97	96	130	140	160	127.3
256	150	150	210	230	250	253.7
384	190	200	290	300	320	379
512	245	248	380	395	500	502
768	360	367	560	620	830	743
1024	800	800	1250	1350	1520	960

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 196

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	13.9	17	30	100	0.996
8	24	21.5	26	30	83	7.96
16	30	28.3	34	40	90	15.9
32	41.7	40	51	57	100	31.7
48	50	47	64	90	200	47.2
64	60	70	75	80	140	63.3
128	100	90	120	200	300	125.7
256	1500	1500	2700	2800	3500	208

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 750

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	20	50	100	100	0.996
64	90	80	200	200	300	63.3
128	100	100	200	300	400	125.6
256	200	170	400	500	700	247
384	220	200	400	500	600	369
512	300	300	500	700	900	480
768	500	400	1000	1000	2000	700
1024	1000	900	2000	2000	2000	890

Language model: n-gram

# of streams	Throughput (RTFX)
32	2260

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 227

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	13.6	14.5	15	30	1
8	19	18.5	20	22	43	8
16	25	23.6	30	32	65.4	15.98
32	35	34	44	46	80	31.9
48	44	43	60.4	62	100	47.8
64	47.6	50	64	66	120	63.8
128	79	66	97	100	350	126.8
256	800	800	1500	1500	2000	240.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 907

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	14.3	15.7	30	100	1
64	70	70	80	150	200	63.6
128	90	80	120	230	320	126.6
256	139	120	175	340	580	251
384	192	173	250	500	860	373
512	270	220	666	700	1350	492
768	520	339	1400	2100	2500	723
1024	960	617	2400	3200	3300	943

Language model: n-gram

# of streams	Throughput (RTFX)
32	510

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 204

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16.2	16	17	17.7	40	1
8	24	23.3	25.5	30	60	7.99
16	30	30	35	37	70	15.97
32	42	40	50	53.5	102	31.9
48	48	48	64	67	120	47.8
64	56	60	72	75	170	63.7
128	97	90	117	160	300	127.2
256	6000	5000	10000	11000	14000	210

Chunk size (ms): 800

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24	21.5	23	40	100	0.999
64	90	90	100	200	400	63.6
128	130	110	200	300	700	126.5
256	300	170	900	2000	2000	248
384	500	230	1000	2000	3000	373
512	700	300	2000	3000	4000	490
768	3000	2000	6000	7000	8000	716

Language model: n-gram

# of streams	Throughput (RTFX)
32	106

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 179

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	15.3	16.2	16.3	40	0.999
8	21.6	20.4	22	23	59	7.99
16	28	26.4	30	39	80	15.96
32	41.4	40	53	54	130	31.85
48	49	54	64	66	160	47.7
64	59	67	75	76	216	63.6

Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 810

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	19.6	30	40	100	0.999
64	90	93	110	200	240	63.5
128	115	100	140	260	350	126.6
256	185	163	248	451	630	251
384	254	230	350	630	930	373
512	362	300	730	940	1550	491

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	300
False	32	2000
True	32	125

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 110

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19	18.5	19.4	20	40	0.999
8	28	27.4	28	30	46.4	7.99
16	38	36	50	52	64	15.96
32	49.4	50	63.3	64	104	31.9
48	68	80	84	86	145	47.8
64	87	101	106	110	188	63.6

Chunk size (ms): 960
Language model: none
Maximum effective # of streams without language model (greedy generation): 555

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	19.9	20	30	100	1
64	100	113.8	134	137	140	63.7
128	156	150	210	215	236	127
256	262	250	396	400	480	252
384	378	370	593	600	860	374
512	520	530	800	980	1200	492

Language model: none

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	300
False	32	2240
True	32	82

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 104

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24	22.7	23.7	25	50	0.999
8	32.7	31	33	51	72.7	7.98
16	44	40.8	50	63	110	15.94
32	59	60	73	75	180	31.8
48	79	90	93	100	240	47.6
64	100	109	114	160	310	63.4

Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 490

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	29.1	40	50	100	0.999
64	123	130	160	240	260	63.5
128	185	165	240	360	430	126.4
256	300	266	430	630	830	249.4
384	460	445	770	1100	1560	368
512	720	650	1400	1550	2150	483

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	180
False	32	1330
True	32	75

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 102

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10000000	10000000	10000000	10000000	10000000	0.998
8	10000000	10000000	10000000	10000000	10000000	7.98
16	10000000	10000000	10000000	10000000	10000000	15.92
32	10000000	10000000	10000000	10000000	10000000	31.8
48	10000000	10000000	10000000	10000000	10000000	47.5
64	10000000	10000000	10000000	10000000	10000000	63.2

Chunk size (ms): 960
Language model: none
Maximum effective # of streams without language model (greedy generation): 490

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10000000	10000000	10000000	10000000	10000000	0.999
64	10000000	10000000	10000000	10000000	10000000	63.5
128	10000000	10000000	10000000	10000000	10000000	126.3
256	10000000	10000000	10000000	10000000	10000000	249.6
384	10000000	10000000	10000000	10000000	10000000	370
512	10000000	10000000	10000000	10000000	10000000	477

Language model: none

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	30
False	32	200

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 141
Maximum effective # of streams without language model (greedy generation): 142

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	13.5	12.5	13.6	14.3	40	0.999
n-gram	8	26	24	30	40	67	7.98
n-gram	16	37	34	44	50	95	15.95
n-gram	32	48.4	47	64	71	130	31.85
n-gram	48	63	64	80	90	200	47.7
n-gram	64	81	86	100	130	300	63.4
none	1	13	12	12.8	13.4	40	0.999
none	8	23	22	30	36	60	7.98
none	16	33	30	40	45	90	15.95
none	32	45	45	60	66	140	31.85
none	48	60	60	75	90	170	47.7
none	64	72	79	93	110	290	63.4

Chunk size (ms): 800
Maximum effective # of streams with n-gram language model: 640
Maximum effective # of streams without language model (greedy generation): 650

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	17	15	16	32	70	0.999
n-gram	64	100	100	120	200	250	63.5
n-gram	128	140	130	170	300	420	126.3
n-gram	256	220	200	290	490	760	250
none	1	15	12.7	14	27.3	70	1
none	64	90	90	107	190	240	63.5
none	128	127	110	160	270	390	126.4
none	256	207	180	270	500	750	250
none	384	310	276	600	700	1300	371

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 143

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	19.4	20	25	90	0.998
8	25.8	24.3	30	40	50	7.98
16	35	33	46	50	70	15.93
32	48	50	65	74	108	31.8
64	80	90	100	120	250	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 510

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	22	60	70	70	0.997
64	107	100	224	240	250	62.9
128	160	133	320	390	420	124
256	257	230	550	700	780	242

Language model: n-gram

# of streams	Throughput (RTFX)
32	225

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 150

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.8	14	15	30	1
8	24.5	23.2	28	33	50	7.99
16	34	33	41.6	46	70	15.97
32	44.4	44	60	64	90	31.9
64	70	84	95	100	200	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 630

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	13.9	20	30	70	1
64	100	102	120	200	250	63.7
128	136	120	170	290	400	127.2
256	218	200	290	520	754	253.3

Language model: n-gram

# of streams	Throughput (RTFX)
32	450

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 134

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	12.6	13.8	20	80	0.997
8	26	23.3	30	40	80	7.95
16	34	30	42	48	90	15.87
32	48	50	62	70	130	31.6
64	80	90	100	140	300	62.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 440

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	14	40	80	80	0.997
64	113	105	230	235	250	62.6
128	166	130	370	420	430	123
256	280	230	600	740	900	236.7

Language model: n-gram

# of streams	Throughput (RTFX)
32	167

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 148

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	19	20	20	30	0.999
8	27	26	31.5	33	44	7.99
16	36	35	42	45	54	15.97
32	47	50	60	63	70	31.9
64	70	80	96	100	110	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 654

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24	21.4	30	37	90	0.999
64	90	102	110	116	120	63.7
128	130	136	167	183	210	127
256	210	205	290	340	450	251.5

Language model: n-gram

# of streams	Throughput (RTFX)
32	400

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 146

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.8	13	13.3	50	1
8	24	23	26	30	70	7.98
16	34	32	40	40	90	15.95
32	46	47	60	64	100	31.86
48	60	60	80	80	150	47.7
64	80	90	100	100	200	63.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 600

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	13.7	14.6	30	70	1
64	100	100	120	230	260	63.5
128	143	120	170	346	480	126
256	238	200	350	700	1120	248.3

Language model: n-gram

# of streams	Throughput (RTFX)
32	1040

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 120

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25	23	28	30	100	0.996
8	40	38	46.6	50	110	7.94
16	50	48	58	62	140	15.8
32	70	70	80	90	250	31.54
48	90	90	100	150	400	47.2
64	105	110	126	160	350	62.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 430

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	40	33	58	70	100	0.996
64	150	130	300	300	400	62.7
128	210	200	400	500	540	123.6
256	360	300	700	1000	1300	240

Language model: n-gram

# of streams	Throughput (RTFX)
32	800

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 153

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.7	13.6	13.8	30	1
8	25.4	24.7	30	30.8	50	7.99
16	34	33	44	50	60	15.96
32	50	50	60	70	80	31.9
64	73	84	95	100	130	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 700

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	14	20	30	70	1
64	86	96	110	120	130	63.7
128	121	114	160	165	190	127
256	190	190	277	284	300	253

Language model: n-gram

# of streams	Throughput (RTFX)
32	440

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 130

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22	19.7	22	22	80	0.996
8	33	30	38	40	100	7.94
16	43.2	40	50	56	130	15.85
32	53	53	67	70	170	31.6
48	80	80	90	140	300	47.1
64	90	100	110	150	300	62.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 470

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	26.7	70	90	90	0.996
64	130	116	300	300	300	62.6
128	200	160	400	500	600	123.3
256	300	260	600	800	1000	239.7

Language model: n-gram

# of streams	Throughput (RTFX)
32	700

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 149

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.7	13.5	13.7	50	0.999
8	24.3	23.3	26	30	70	7.98
16	33	32	40	42	90	15.95
32	46	47	61.6	65	110	31.84
64	77	86	95	100	230	63.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 630

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	13.7	20	50	70	0.999
64	98	101	120	200	260	63.5
128	140	120	170	330	430	126
256	217	196	282	530	740	249

Language model: n-gram

# of streams	Throughput (RTFX)
32	440

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 123

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	19.3	30	40	70	0.994
8	28	24.5	50	52	55	7.95
16	38	34	56	70	80	15.8
32	50	50	80	100	115	31.54
64	90	90	150	180	250	62.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 400

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	20.7	90	90	90	0.994
64	130	100	240	240	250	61.7
128	198	157	400	420	430	120.3
256	330	260	700	750	800	229

Language model: n-gram

# of streams	Throughput (RTFX)
32	110

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 112

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	20	40	40	90	0.991
8	32	27	64	70	76	7.88
16	41	34.5	70	100	105	15.6
32	55	50	90	140	150	30.9
64	110	90	260	300	370	60

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 300

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	40	40	90	90	90	0.99
64	155	110	260	260	260	60.3
128	240	170	420	450	450	116.7
256	400	300	770	800	840	215

Language model: n-gram

# of streams	Throughput (RTFX)
32	71.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 140

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	12.6	14	20	70	0.997
8	26	23.3	30	33	70	7.96
16	33	30	41.5	50	95	15.87
32	48	50	63	70	140	31.6
64	80	84	100	200	300	62.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 476

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	14	50	70	70	0.997
64	120	106	230	260	270	62.5
128	164	140	340	386	420	123.2
256	267	232	570	700	800	239

Language model: n-gram

# of streams	Throughput (RTFX)
32	180

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 140

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.7	13.7	16	40	1
8	25	23	30	40	66	7.98
16	34	32	43	49	90	15.95
32	45	45	60	67	130	31.84
48	60	60	79	90	200	47.7
64	78	82	98	122	300	63.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 547

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	13.2	20	40	70	1
64	110	100	226	235	250	63.5
128	167	143	370	392	410	126.3
256	268	230	600	680	770	249.6

Language model: n-gram

# of streams	Throughput (RTFX)
32	1080

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 139

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	13	14	20	70	0.997
8	27	24	30	30	76	7.95
16	36	32	43	50	106	15.87
32	49	47	64	74	140	31.6
48	65	70	83	90	220	47.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 520

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	17	50	70	70	0.997
64	120	105	250	270	300	62.5
128	170	140	370	420	430	123.4
256	270	220	600	730	790	239.6

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	192

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 124

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22	19.27	25	40	100	0.997
8	30	26	50	66	80	7.95
16	40	35	70	86	102	15.86
32	54	50	85	116	145	31.6
48	70	70	100	170	230	47.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 505

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	23	60	80	80	0.997
64	120	108	250	260	270	62.6
128	170	140	370	410	425	123.4
256	275	235	600	760	800	239.5

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	192

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 14

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	26	25	26	27	50	0.998
4	56.8	55.3	61.1	65	108	3.99
8	84.6	95.5	103.4	106	174	7.95

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 68

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	26.9	28	44	80	0.999
4	60	58	64	84	100	3.99
8	87.6	97	110	116	146	7.97

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	130
False	4	202
False	8	208

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 150

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	19.56	21	22	40	0.999
8	27	26	31	33	49	7.99
16	35.5	35	44	47	60	15.96
32	47	50	63	65	76	31.9
48	57	60	77	80	90	47.8
64	80	90	100	100	110	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 650

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24	22	27	40	90	0.999
64	92	104	120	120	130	63.7
128	130	125	170	190	210	127
256	210	200	295	350	459	251.4

Language model: n-gram

# of streams	Throughput (RTFX)
32	160

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 135

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	21.7	21	23	27	40	0.999
8	35	36	38	40	54	7.99
16	44	44	50	52	60	15.96
32	56	60	68	70	80	31.9
48	67	70	85	90	100	47.8
64	81.5	96	107	110	130	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 600

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25	23	30	35	90	0.999
64	110	120	140	140	140	63.7
128	152	145	196	200	240	127
256	230	220	330	340	360	252.5

Language model: n-gram

# of streams	Throughput (RTFX)
32	1400

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 130

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24	21.3	24.5	26	100	0.996
8	36	36	40	46	90	7.95
16	46	45	53	60	110	15.86
32	57	56	70	80	160	31.6
48	73	80	90	100	200	47.3
64	85	95	110	118	200	63

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 553

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	26	60	80	80	0.996
64	100	110	200	200	300	63.1
128	170	150	300	400	500	125
256	250	240	400	500	600	246

Language model: n-gram

# of streams	Throughput (RTFX)
32	1600

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 144

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	19.5	21	23	40	1
8	27	26	31	33	50	7.99
16	38	37	45	50	80	15.98
32	49	50	60	64	120	31.9
64	80	90	100	100	220	63.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 610

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24	22	29	39	90	0.999
64	98	102	114	190	250	63.5
128	143	125	172	300	400	126.3
256	220	200	300	500	800	249.4

Language model: n-gram

# of streams	Throughput (RTFX)
32	500

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 140

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24	23	26	30	50	1
8	36	34.6	42	45	80	7.98
16	44.5	43.6	52	56	100	15.95
32	60	60	75	80	160	31.9
64	90	100	110	120	200	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 550

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	32	29.3	31	50	100	0.998
64	130	120	200	200	400	63.5
128	200	160	300	500	900	126
256	500	250	1000	2000	3000	250

Language model: n-gram

# of streams	Throughput (RTFX)
32	100

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 111

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25	24	26	27	50	0.998
8	33	31	40	48	80	7.98
16	43	40	53	57	107	15.94
32	60	60	78	83	170	31.8
48	80	84	100	110	250	47.6

Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 514

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	27	40	50	90	0.999
64	136	140	170	270	280	63.4
128	192	160	244	380	500	126.1
256	310	300	420	700	940	248.6

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	180
False	32	1300
True	32	120

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 61

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	28.4	27.6	28.4	30	53	0.998
8	40	38.4	50	50	70	7.98
16	55.4	52	72.4	76	90	15.94
32	90	104	114	117	180	31.8
48	122	138	150	155	260	47.6
64	240	220	370	400	430	63.3

Chunk size (ms): 960
Language model: none
Maximum effective # of streams without language model (greedy generation): 324

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	29.5	37	50	100	0.999
64	188	203	214	218	220	63.6
128	274	240	380	385	400	126.4
256	480	500	710	780	1060	247.6
384	2500	2400	4200	4500	5000	330

Language model: none

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	200
False	32	1470
True	32	116

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 65

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	33.5	32.6	33.7	36	60	0.998
8	47	43	60	69	100	7.97
16	63	60	82	90	140	15.92
32	96	106	118	130	260	31.74
48	139	145	160	220	380	47.4

Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 320

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	40	37	50	66	100	0.998
64	203	204	220	360	380	63.3
128	284	260	373	510	630	125.5
256	540	500	1000	1260	1900	244.6

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	130
False	32	850
True	32	104

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 58

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10000000	10000000	10000000	10000000	10000000	0.998
8	10000000	10000000	10000000	10000000	10000000	7.97
16	10000000	10000000	10000000	10000000	10000000	15.9
32	10000000	10000000	10000000	10000000	10000000	31.6
48	10000000	10000000	10000000	10000000	10000000	47.2
64	10000000	10000000	10000000	10000000	10000000	59.4

Chunk size (ms): 960
Language model: none
Maximum effective # of streams without language model (greedy generation): 284

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10000000	10000000	10000000	10000000	10000000	0.998
64	10000000	10000000	10000000	10000000	10000000	63.2
128	10000000	10000000	10000000	10000000	10000000	125.1
256	10000000	10000000	10000000	10000000	10000000	242.3
384	10000000	10000000	10000000	10000000	10000000	282

Language model: none

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	19
False	32	120

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 141
Maximum effective # of streams without language model (greedy generation): 141

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	17	15.7	16.7	17.4	40	0.999
n-gram	8	34	30	40	47	80	7.98
n-gram	16	45	42	55	60	120	15.94
n-gram	32	55	54	74	80	200	31.8
n-gram	48	65	70	83	93	230	47.65
n-gram	64	83	88	100	130	300	63.4
none	1	16	15	15.6	16	40	0.999
none	8	30.8	29	33	44	75	7.98
none	16	42	39	53	57	110	15.94
none	32	53	50	70	76	160	31.8
none	48	59	60	77	86	200	47.7
none	64	80	84	96	110	270	63.4

Chunk size (ms): 800
Maximum effective # of streams with n-gram language model: 625
Maximum effective # of streams without language model (greedy generation): 630

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	20	17.7	19	36	70	0.999
n-gram	64	110	113	140	245	300	63.5
n-gram	128	140	120	170	300	407	126.3
n-gram	256	223	200	295	500	770	250
none	1	20	15.6	16.5	31	70	1
none	64	105	107	136	220	250	63.5
none	128	133	110	165	290	400	126.4
none	256	212	190	284	500	730	250
none	384	310	290	600	640	1300	371.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 140

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24	22	26	30	90	0.997
8	34.6	33	40	52	63	7.97
16	46	43	60	65	90	15.9
32	57	55	80	85	130	31.8
64	78	88	100	110	200	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 514

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	24	50	90	90	0.997
64	125	120	250	270	300	62.7
128	160	130	300	400	400	124.2
256	260	223	530	700	780	242.3

Language model: n-gram

# of streams	Throughput (RTFX)
32	224

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 147

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	15.3	16.4	18	40	1
8	32	30.3	40	40	61	7.99
16	43	40	50	57	80	15.96
32	54	55	73	76	100	31.9
64	76	87	97	102	200	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 630

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	16.2	23	40	70	1
64	110	120	150	240	300	63.7
128	140	127	200	280	400	127.2
256	220	200	295	500	700	253.3

Language model: n-gram

# of streams	Throughput (RTFX)
32	460

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 136

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	15.2	16.3	20	70	0.997
8	32	30	35	40	80	7.94
16	42	38.3	50	60	109	15.84
32	56	55	75	83	160	31.54
64	80	88	102	130	280	62.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 450

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	17	50	70	70	0.997
64	126	120	250	260	280	62.4
128	166	140	300	390	400	123.2
256	272	236	570	700	780	237.6

Language model: n-gram

# of streams	Throughput (RTFX)
32	167

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 145

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23	22	24	25	40	0.999
8	33.5	32.6	38	40	55	7.99
16	46	45	58	60	70	15.96
32	56	60	75	78	87	31.9
64	76	90	100	105	110	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 650

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	24.7	30	40	90	0.999
64	110	120	140	150	150	63.7
128	136	132	177	200	216	127
256	210	208	300	360	460	251.3

Language model: n-gram

# of streams	Throughput (RTFX)
32	400

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 145

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	15.5	16.4	16.5	50	0.999
8	34	31.6	40	44	80	7.98
16	43	41	54.6	56	100	15.94
32	54	52.4	70.9	74	120	31.84
48	59	60	75	78	150	47.7
64	76	87	96	99	200	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 600

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	16.14	17	32	70	0.999
64	110	113	150	240	300	63.5
128	150	130	186	360	480	126
256	240	210	300	670	1000	248.4

Language model: n-gram

# of streams	Throughput (RTFX)
32	1060

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 118

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	28	26	29	30	100	0.996
8	45	42	50	60	120	7.94
16	57	56	68	73	155	15.83
32	68	70	85	100	250	31.5
48	90	94	105	140	310	47.2
64	110	113	130	160	370	62.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 450

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	40	35	60	72	100	0.996
64	150	130	300	300	400	62.7
128	200	194	400	500	500	124
256	300	290	700	800	1000	241

Language model: n-gram

# of streams	Throughput (RTFX)
32	1000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 144

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	15.6	16.6	16.9	30	0.999
8	33	32	36	38	58	7.99
16	44	42	54	57	70	15.95
32	55	60	74	76	90	31.9
64	80	90	100	104	140	63.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 660

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	16.5	20	30	70	0.999
64	110	120	146	150	160	63.6
128	130	125	170	180	200	127
256	196	190	286	290	310	253

Language model: n-gram

# of streams	Throughput (RTFX)
32	440

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 120

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	26	24	28	32	100	0.995
8	40.5	37	50	50	120	7.93
16	52	50	60	66	150	15.84
32	60	60	76	86	180	31.56
48	80	86	100	140	300	47.1
64	96	100	120	200	300	62.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 480

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	40	28	60	100	100	0.996
64	150	130	300	400	400	62.7
128	200	160	360	500	600	123.8
256	300	260	600	700	1000	241

Language model: n-gram

# of streams	Throughput (RTFX)
32	700

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 147

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	15.6	16.3	16.5	50	0.999
8	32.4	31	35	36	80	7.98
16	44	42	54	57	100	15.93
32	56	60	73	77	130	31.8
64	80	90	99	102	240	63.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 620

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	17	20	50	80	0.999
64	120	130	150	250	300	63.3
128	140	130	200	300	400	126
256	226	200	294	500	740	249

Language model: n-gram

# of streams	Throughput (RTFX)
32	447

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 123

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25	22	30	40	90	0.994
8	35.4	31.2	56	60	64	7.93
16	48	42	70	90	103	15.83
32	58	60	90	120	150	31.5
64	90	91	166	200	260	62.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 400

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	40	26	90	90	90	0.994
64	150	110	300	300	300	61.4
128	200	160	400	400	400	120.5
256	320	260	660	720	750	230

Language model: n-gram

# of streams	Throughput (RTFX)
32	110

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 110

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	22.5	40	50	100	0.99
8	40	33.5	70	90	100	7.86
16	50	43	90	110	120	15.58
32	60	60	90	140	160	31
64	110	100	200	330	350	60.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 330

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	40	41	90	90	90	0.99
64	160	130	260	260	260	60.6
128	230	180	400	400	400	117
256	398	320	750	750	760	217

Language model: n-gram

# of streams	Throughput (RTFX)
32	71.6

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 136

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	16.3	18	20	70	0.997
8	34	31	40	50	90	7.94
16	47	44	58	60	120	15.83
32	60	60	80	90	160	31.5
64	87	93	107	140	260	62.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 480

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	17	50	70	70	0.997
64	130	110	240	300	300	62.4
128	166	130	350	370	400	123
256	267	240	580	700	750	239

Language model: n-gram

# of streams	Throughput (RTFX)
32	177

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 140

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	15.2	16.2	20	40	0.999
8	32	30	40	47	78	7.98
16	43	40	55	59	120	15.93
32	54	50	70	80	170	31.8
48	63	70	82	90	200	47.7
64	80	87	100	116	300	63.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 550

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	16.3	30	40	70	0.999
64	130	110	260	300	300	63.4
128	165	140	310	374	400	126.3
256	269	230	570	670	730	250

Language model: n-gram

# of streams	Throughput (RTFX)
32	1100

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 138

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	16.3	17.6	20	80	0.997
8	35	32.5	38	40	90	7.94
16	47	44	60	60	120	15.84
32	57	57	80	84	150	31.57
48	70	73	86	100	240	47.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 520

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	20	60	80	80	0.996
64	140	130	280	290	300	62.4
128	170	150	340	380	400	123.7
256	270	235	580	700	760	240

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	191

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 127

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25	22.5	28	45	100	0.996
8	37	32	58	72	80	7.94
16	49	42	80	100	120	15.84
32	58	55	90	120	150	31.6
48	72	70	100	160	230	47.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 500

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	28	50	100	100	0.996
64	140	120	300	300	300	62.5
128	170	140	340	400	410	123.7
256	270	240	600	700	760	240.6

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	192

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 7

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	54	53	60	60	100	0.997
4	98	95	101	110	190	3.98
8	1360	1310	2440	2540	2930	7.37

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 36

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	70	68.8	70	86.7	100	0.997
4	110	100	140	140	150	3.984
8	153	170	185	200	230	7.94

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	70
False	4	106
False	8	109.3

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 145

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23	22.4	23.5	24	40	0.999
8	34	33	38	40	53	7.98
16	46	45	57	60	65	15.95
32	57	60	76	79	85	31.9
48	61	64	80	85	93	47.8
64	80	90	100	105	114	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 630

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	27	24.3	30	42	90	0.999
64	107	120	140	150	160	63.6
128	136	130	176	200	214	127
256	210	210	300	360	460	251.4

Language model: n-gram

# of streams	Throughput (RTFX)
32	146

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 134

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23	23	24.5	26	40	0.999
8	40	39.5	50	50	58.7	7.98
16	52	52	59	60	70	15.96
32	60	60	76	80	90	31.9
48	70	73	88	92	100	47.8
64	83	100	108	110	120	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 600

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	26	34	40	90	0.999
64	120	130	150	160	170	63.6
128	155	150	200	220	240	127
256	234	230	336	340	370	252.3

Language model: n-gram

# of streams	Throughput (RTFX)
32	1500

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 130

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	27	24.4	28	33	100	0.995
8	44	42	47	49	90	7.95
16	54	53	63	70	110	15.86
32	62	63	77	85	140	31.6
48	80	80	90	100	200	47.25
64	90	100	110	120	200	63.1

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 500

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	40	30	70	100	100	0.995
64	140	130	200	200	300	63
128	180	160	300	300	500	125
256	260	251	400	500	700	247

Language model: n-gram

# of streams	Throughput (RTFX)
32	1700

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 143

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23	22.8	24	27	34	0.999
8	35	34	39	40	65	7.99
16	47	45	55	60	100	15.97
32	60	60	80	80	120	31.9
64	80	90	100	105	230	63.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 600

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	27	24.6	30	44	90	0.999
64	120	120	150	230	300	63.4
128	146	130	180	300	410	126.3
256	230	200	300	560	800	249.5

Language model: n-gram

# of streams	Throughput (RTFX)
32	500

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 140

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25	24.6	30	30	47	0.999
8	42	41	47	50	80	7.98
16	54.5	54	63	67	110	15.95
32	64	67	81	86	150	31.9
64	90	100	110	120	200	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 530

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	34	31.7	35	50	100	0.998
64	140	134	200	200	400	63.5
128	200	160	300	400	1000	125.6
256	500	260	1000	2000	3000	250

Language model: n-gram

# of streams	Throughput (RTFX)
32	100

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 109

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	29	28	29	33	60	0.998
8	43	40	50	60	95	7.97
16	53	50	70	70	140	15.93
32	60	60	76	80	160	31.8
48	80	88	98	104.4	250	47.6

Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 490

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	34	31.5	40	50	100	0.998
64	140	144	170	260	300	63.4
128	200	170	260	370	470	126.1
256	325	300	440	722	950	248.6

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	200
False	32	1280
True	32	123

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 59

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30.5	30	31	33	55	0.998
8	51	50	55	56	90	7.98
16	58	56	70	75	100	15.94
32	86.4	90	110	114	180	31.8
48	123	144	155	170	260	47.6
64	710	700	1160	1200	1230	61.4

Chunk size (ms): 960
Language model: none
Maximum effective # of streams without language model (greedy generation): 320

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	36	33	40	53	100	0.999
64	184	200	214	220	240	63.5
128	280	250	386	390	420	126.4
256	496	500	730	830	1100	247.4
384	2930	2770	5020	5500	6200	322.6

Language model: none

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	200
False	32	1450
True	32	126

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 64

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	40	39.2	40	45	76	0.998
8	57	56	62	80	120	7.97
16	63	60	81	87	150	15.92
32	94	100	117	130	200	31.75
48	135	145	163	230	360	47.5

Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 304

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	50	45	60	70	100	0.998
64	205	214	226	300	350	63.3
128	310	260	400	520	643	125.5
256	580	560	1030	1330	2000	244.6

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	150
False	32	825
True	32	84

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 55

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10000000	10000000	10000000	10000000	10000000	0.997
8	10000000	10000000	10000000	10000000	10000000	7.97
16	10000000	10000000	10000000	10000000	10000000	15.9
32	10000000	10000000	10000000	10000000	10000000	31.6
48	10000000	10000000	10000000	10000000	10000000	47.2
64	10000000	10000000	10000000	10000000	10000000	56.6

Chunk size (ms): 960
Language model: none
Maximum effective # of streams without language model (greedy generation): 276

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10000000	10000000	10000000	10000000	10000000	0.998
64	10000000	10000000	10000000	10000000	10000000	63.2
128	10000000	10000000	10000000	10000000	10000000	125
256	10000000	10000000	10000000	10000000	10000000	241.3
384	10000000	10000000	10000000	10000000	10000000	275

Language model: none

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	27
False	32	108

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 110
Maximum effective # of streams without language model (greedy generation): 112

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	14	13	13.6	14	40	0.999
n-gram	8	27	25	30.6	37	70	7.98
n-gram	16	39	37	47	51	100	15.94
n-gram	32	52	50	70	74	150	31.8
n-gram	48	70	70	90	100	250	47.6
n-gram	64	100	100	120	200	350	63.3
none	1	12.6	12	12.54	12.7	40	0.999
none	8	24	22.9	26	29	60	7.98
none	16	33.6	31	40	44	90	15.95
none	32	47	47	62	70	140	31.8
none	48	63	70	83	90	230	47.6
none	64	87	94	106	140	325	63.4

Chunk size (ms): 800
Maximum effective # of streams with n-gram language model: 510
Maximum effective # of streams without language model (greedy generation): 500

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	17	14.9	16	32	80	0.999
n-gram	64	110	114	130	240	300	63.4
n-gram	128	164	150	200	370	470	126.1
n-gram	256	270	255	360	600	850	249
none	1	15	12.6	13	26.6	70	1
none	64	100	104	120	200	260	63.5
none	128	152	130	192	300	450	126.2
none	256	263	240	355	550	900	249
none	384	430	377	800	1250	1800	369

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 110

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	19.4	22	26	60	0.998
8	29	27	40	47	55	7.97
16	39	38	48	58	80	15.92
32	52	52	67	75	120	31.8
64	90	100	116	150	310	63.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 425

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	21	60	70	70	0.997
64	120	116	250	260	270	62.8
128	190	160	370	440	466	123.8
256	323	290	640	800	1000	240

Language model: n-gram

# of streams	Throughput (RTFX)
32	225

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 113

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.7	14	14.5	30	1
8	26	25	30	34	53	7.99
16	36	35	44	50	80	15.97
32	50	50	67	70	94	31.9
64	90	100	110	120	200	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 501

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	14	20	30	70	1
64	106	110	130	200	260	63.7
128	160	140	200	340	470	127
256	270	253	367	613	900	252.7

Language model: n-gram

# of streams	Throughput (RTFX)
32	457

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 107

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	12.8	13.5	20	70	0.997
8	27	24	30	30	80	7.95
16	36	33	43	50	94	15.86
32	51	50	70	80	140	31.56
64	95	100	110	200	350	62.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 374

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	14	50	70	70	0.997
64	120	112	240	250	260	62.4
128	190	160	400	440	460	122.4
256	366	300	800	940	1200	233.7

Language model: n-gram

# of streams	Throughput (RTFX)
32	166

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 116

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	19.5	20	25	40	0.999
8	27	26.3	30	34	46	7.99
16	36	35.5	45	47	55	15.96
32	50	50	67	70	80	31.9
64	83	100	110	113	122	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 520

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24	21	30	30	90	0.999
64	100	117	130	130	140	63.7
128	158	150	207	220	280	126.6
256	260	250	370	460	600	250

Language model: n-gram

# of streams	Throughput (RTFX)
32	427

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 112

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	12.9	13.6	13.9	40	0.999
8	27	25.4	30	30	74	7.98
16	35.74	34	44	46	80	15.95
32	50	50	70	70	111.7	31.85
48	70	79	90	94	200	47.7
64	87	100	110	114	260	63.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 490

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	14.2	15	29	80	1
64	103.4	110	120	200	270	63.5
128	170	140	200	420	550	125.8
256	310	250	630	800	1400	247

Language model: n-gram

# of streams	Throughput (RTFX)
32	930

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 97

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	26	24.6	28	32	100	0.996
8	41.7	40	47	50	114	7.94
16	50	49	60	65	140	15.84
32	70	70	85	90	220	31.5
48	90	94	110	150	340	47.2
64	113	120	140	200	400	62.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 370

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	40	32	55	60	100	0.996
64	150	140	300	300	400	62.4
128	230	230	500	600	600	123.6
256	440	370	900	1100	1500	239

Language model: n-gram

# of streams	Throughput (RTFX)
32	860

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 116

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.3	13	14	14	30	0.999
8	25	25	28	30	50	7.99
16	34.8	34	40	44	60	15.96
32	50	50	60	70	80	31.9
64	80	98	106	110	170	63.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 550

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	13.8	20	30	40	0.999
64	97	109	125	130	140	63.7
128	145	130	193	200	230	127
256	233.5	230	346	350	380	252.4

Language model: n-gram

# of streams	Throughput (RTFX)
32	440

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 104

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23	21	24	30	100	0.996
8	36	33	40	43	100	7.94
16	46	44	52	56	130	15.84
32	63	65	80	90	180	31.6
48	85	90	101	160	400	47
64	100	110	120	200	370	62.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 393

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	25.6	60	100	100	0.996
64	150	133	300	300	400	62.5
128	210	170	400	500	600	123
256	400	338	700	900	1300	239

Language model: n-gram

# of streams	Throughput (RTFX)
32	700

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 114

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	12.9	13.8	14	40	0.999
8	27	26	30	32	74	7.98
16	36	35	40	50	100	15.94
32	51	54	70	73	120	31.8
64	87	100	110	114	270	63.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 500

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	14	20	50	70	0.999
64	107	111	130	230	270	63.4
128	162	139	200	340	440	126
256	270	252	360	620	900	248

Language model: n-gram

# of streams	Throughput (RTFX)
32	440

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 100

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22	19	30	40	70	0.994
8	31	27	50	54	60	7.94
16	42	38	64	80	90	15.8
32	57	56	90	110	130	31.5
64	110	105	200	250	330	61.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 341

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	21	90	90	90	0.994
64	150	118	300	300	300	61.3
128	226	190	430	460	480	119.5
256	410	340	800	900	1000	226

Language model: n-gram

# of streams	Throughput (RTFX)
32	109.2

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 90

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24	20	37	50	80	0.991
8	35	30	60	73	80	7.88
16	45	40	60	100	100	15.64
32	60	60	90	160	170	30.8
64	136	110	320	440	450	59.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 270

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	40	40	80	80	80	0.99
64	170	130	270	270	270	60.1
128	270	205	470	483	484	115
256	500	440	900	1000	1060	212

Language model: n-gram

# of streams	Throughput (RTFX)
32	71.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 111

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	13	15	20	70	0.997
8	28.5	26	33	40	73	7.95
16	37.4	34	46	53	100	15.86
32	54	60	70	80	150	31.55
64	90	98	110	170	330	62.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 390

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	14.3	40	80	80	0.997
64	123	115	250	260	270	62.4
128	190	150	400	440	450	122.8
256	330	290	660	800	1000	236.7

Language model: n-gram

# of streams	Throughput (RTFX)
32	178

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 111

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12	13	15	40	0.999
8	27	24	30	40	70	7.98
16	35	33	40	50	90	15.94
32	50	50	64	70	140	31.8
48	70	70	87	110	250	47.6
64	87	90	110	140	400	63.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 443

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	13.8	20	40	70	0.999
64	118	108	240	250	260	63.5
128	188	170	370	430	450	126
256	344	290	700	800	1000	248.6

Language model: n-gram

# of streams	Throughput (RTFX)
32	940

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 110

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	13	14	20	70	0.997
8	30	27	40	40	80	7.94
16	39	36	47	53	105	15.86
32	53	54	70	80	150	31.6
48	75	80	90	110	280	47.1

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 427

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	16	40	80	80	0.997
64	130	115	260	280	300	62.5
128	194	160	400	450	460	123
256	335	290	670	800	1000	238

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	190

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 103

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22	19.7	25	40	100	0.996
8	31	26	49	66	80	7.95
16	40.5	35	70	88	104	15.86
32	56	55	80	110	140	31.6
48	78	77	120	200	250	47.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 420

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	24	50	90	90	0.996
64	134	119	284	300	300	62.4
128	191	150	400	430	460	123
256	340	300	700	850	1000	237.7

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	191

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 9

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	36	35	36	36.6	68	0.998
4	71.3	69	75	80	150	3.98
8	118.2	135	143	160	248	7.93

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 46

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	40	35.6	37	52.7	100	0.998
4	74	72	80	100	113	3.99
8	115	130	143.4	160	180	7.96

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	100
False	4	135
False	8	137

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 113

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	19.4	21	24	36	0.999
8	28	27	32	35	45	7.99
16	37	37	45	48	53	15.97
32	50	50	64	67	75	31.9
48	65	67	87	90	100	47.8
64	80	100	110	114	120	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 520

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23	21	28	30	60	0.999
64	97	113	128	130.4	140	63.7
128	160	150	210	230	300	126.7
256	260	240	360	460	590	250

Language model: n-gram

# of streams	Throughput (RTFX)
32	130

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 107

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22	21.3	23	26	40	0.999
8	36.2	36.4	40	42	53	7.99
16	45	45	50	53.2	62	15.96
32	58	63	71.7	75	84	31.9
48	73	80	93	98	108	47.8
64	100	110	130	130	150	63.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 493

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	26	23.5	30	40	90	0.999
64	110	120	140	150	155	63.6
128	180	160	230	240	300	127
256	280	265	400	420	500	251

Language model: n-gram

# of streams	Throughput (RTFX)
32	1400

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 105

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23	21	24	30	100	0.996
8	37	36	41	44	84	7.95
16	46	45	54	60	100	15.9
32	60	64	80	83	140	31.64
48	77	80	100	100	200	47.3
64	93	110	118	124	200	63.1

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 456

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	26	60	100	100	0.996
64	130	127	200	200	200	63.1
128	200	170	300	400	500	124.8
256	300	280	400	500	600	246

Language model: n-gram

# of streams	Throughput (RTFX)
32	1400

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 113

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	20	21.5	24	30	0.999
8	28	27.5	32	34	55	7.99
16	38.6	37.7	45	49	80	15.97
32	52.6	54	68	73	110	31.9
64	90	100	110	116	300	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 485

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24	22	26	39	90	0.999
64	106	110	130	200	270	63.4
128	163	140	201	350	476	126
256	296	270	500	720	1200	248

Language model: n-gram

# of streams	Throughput (RTFX)
32	500

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 107

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23.9	22.95	27	30	47	1
8	38.5	38	43	46	80	7.98
16	47	46.5	53	56	100	15.96
32	60	60	75	80	160	31.9
64	100	117	127	130	210	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 450

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	32	29	35	50	100	0.998
64	130	135	200	200	400	63.4
128	220	180	300	400	1000	125.7
256	400	300	700	1000	2000	249

Language model: n-gram

# of streams	Throughput (RTFX)
32	100

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 81

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23.6	22.7	24	27	50	0.998
8	33	30.6	40	45	80	7.98
16	43.9	42	56	60	120	15.94
32	65	72	82	86	200	31.8
48	96	105	120	140	306	47.5

Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 390

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	28	40	50	100	0.999
64	160	167	190	300	340	63.4
128	245	200	316	440	560	125.8
256	446	420	780	1000	1700	246.3

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	170
False	32	1000
True	32	85

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 44

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	29	30	35	56	0.998
8	44.2	42.4	50	60	73	7.98
16	63	60	80	85	110	15.93
32	106	125	136	140	200	31.76
48	1000	1000	1700	1800	2000	45.2
64	6700	6000	13000	14400	16700	45.4

Chunk size (ms): 960
Language model: none
Maximum effective # of streams without language model (greedy generation): 227

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	35	32.2	40	50	100	0.999
64	245	263	280	280	290	63.4
128	390	340	510	520	670	125.4
256	1700	1700	2800	3000	3500	230
384	10900	9800	20500	22000	25000	229

Language model: none

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	170
False	32	1160
True	32	85

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 47

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	31	30.3	32	37	62.7	0.998
8	46	43	50	70	100	7.97
16	63	59	80	84	155	15.92
32	110	120	130	150	300	31.7
48	400	400	700	700	900	46.7

Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 231

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	42	39	50	60	100	0.998
64	260	273	285	350	430	63.2
128	400	340	510	760	880	124.7
256	1400	1400	2300	2400	3000	234

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	120
False	32	640
True	32	95

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 41

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10000000	10000000	10000000	10000000	10000000	0.997
8	10000000	10000000	10000000	10000000	10000000	7.96
16	10000000	10000000	10000000	10000000	10000000	15.87
32	10000000	10000000	10000000	10000000	10000000	31.5
48	10000000	10000000	10000000	10000000	10000000	42
64	10000000	10000000	10000000	10000000	10000000	41.8

Chunk size (ms): 960
Language model: none
Maximum effective # of streams without language model (greedy generation): 208

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10000000	10000000	10000000	10000000	10000000	0.998
64	10000000	10000000	10000000	10000000	10000000	63
128	10000000	10000000	10000000	10000000	10000000	124.2
256	10000000	10000000	10000000	10000000	10000000	205
384	10000000	10000000	10000000	10000000	10000000	206

Language model: none

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	18
False	32	92

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 338
Maximum effective # of streams without language model (greedy generation): 340

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	11	10.2	11.4	13	40	0.999
n-gram	8	13.3	12.2	13.5	20	44	7.99
n-gram	16	17.5	15	22	27	67	15.96
n-gram	32	23.5	20	30	34	90	31.9
n-gram	48	30	30	40	44	100	47.8
n-gram	64	35.6	36	46	48.4	130	63.7
n-gram	128	55	47.6	67	100	230	127
n-gram	256	110	80.8	250	400	560	252
none	1	10	9.7	10.5	11	30	1
none	8	13	11.5	15	20	40	7.99
none	16	16	14	20	24	50	15.97
none	32	23	23	30	33	85	31.9
none	48	28	27	37	41	90	47.8
none	64	32.6	31.4	44	46	104	63.7
none	128	50	44	62	90	200	127
none	256	100.8	76	190	345	500	252

Chunk size (ms): 800
Maximum effective # of streams with n-gram language model: 1430
Maximum effective # of streams without language model (greedy generation): 1430

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	15	11.8	13	28	80	1
n-gram	64	43	44	50	100	145	63.7
n-gram	128	60	50	70	160	230	127
n-gram	256	92	80	110	240	410	252.7
n-gram	384	121	106	153	350	580	377
n-gram	512	148	130	194	400	730	499
n-gram	768	233	187	570	660	1200	740
n-gram	1024	330	240	684	1240	1600	977
none	1	10	9.8	11	23	80	1
none	64	37	40	47	90	120	63.8
none	128	56	47	68	150	220	127
none	256	86	75	107	240	400	253
none	384	114	101	148	300	550	377
none	512	142	120	190	400	700	500
none	768	220	176	510	590	1180	742
none	1024	330	236	700	1200	1700	978

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 344

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10	10.5	12	16	100	0.998
8	14	12.6	20	26	33	7.98
16	19	17	25	34	40	15.95
32	25	25	35	40	60	31.9
48	29	28.7	40	46	80	47.8
64	37	42	48	51	100	63.7
128	52	48	66.6	76	156	127.2
256	100	80	170	370	480	253

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1100

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	11.5	30	100	100	0.998
64	50	44	100	130	134	63.4
128	71	50	160	220	226	126
256	110	83	270	330	400	248.7
384	147.7	113	360	470	570	368
512	182	137.4	460	620	740	485
768	303	210	630	900	1280	710
1024	480	310	1200	1340	1900	924

Language model: n-gram

# of streams	Throughput (RTFX)
32	227

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 355

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10	9.9	11.3	12	40	1
8	12.6	12	13.4	17	31	8
16	17	15	22	25	40	15.98
32	23	23	31	33	50	31.94
48	29	28	40	41	70	47.9
64	33.6	38	45	47	70	63.9
128	49	47	64	67	150	127.6
256	84	75	107	126	391	255

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1400

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	11	20	40	80	1
64	39	40	55	80	110	63.9
128	58	50	75	150	202	127.6
256	90	80	115	240	380	255
384	120	107	155	316	530	381.4
512	149	130	196	400	700	508
768	258	200	630	680	1280	756
1024	420	263	1280	1350	1900	992

Language model: n-gram

# of streams	Throughput (RTFX)
32	467

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 320

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10	10	11	20	80	0.997
8	14	12	15	20	48	7.97
16	17.6	14.6	22	27	60	15.9
32	24	23	32	36	87	31.7
48	30	30	40	43	107	47.5
64	35	40	45	48	123	63.2
128	52	44	65	100	220	125.3
256	116	79	270	430	600	245.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 960

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	12	50	80	80	0.997
64	50	40	110	130	135	63.2
128	73	50	170	200	220	125.3
256	113	81	290	360	400	246.3
384	155	110	400	520	590	362.4
512	193	140	500	640	750	476
768	360	226	800	1100	1350	680
1024	630	350	1400	1600	2100	880

Language model: n-gram

# of streams	Throughput (RTFX)
32	169

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 368

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	10.7	12	12.8	50	1
8	14	12.8	16	20	30	7.99
16	16.9	15.2	22.4	24	30	15.98
32	23	25	32	34	40	31.94
48	28	28.3	38	40	50	47.9
64	34	40	46	47.7	55	63.8
128	47	46	63	65	73	127.6
256	71	73	103	105	110	254.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1590

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10	12	20	20	100	1
64	37	40	50	55	62	63.8
128	50	50	66	70	80	127.6
256	74	76	105	110	124	254.6
384	100	103	147	155	203	381
512	126	126	190	216	290	506
768	180	174	270	350	470	753
1024	233	225	360	490	650	996

Language model: n-gram

# of streams	Throughput (RTFX)
32	434

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 357

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	10.4	11.3	12	50	1
8	13	12	15	20	47	7.99
16	17	15	21	22	50	15.97
32	23	23	30	33	70	31.9
48	29	28	39	41	90	47.8
64	34	40	45	46.5	110	63.7
128	50	46	64	66	160	127
256	84	75	104	110	480	252.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1370

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	11.4	13	25.4	80	1
64	39.2	40	48	94	130	63.7
128	57.5	48.5	69	160	224	127
256	90	77	107	270	420	252.5
384	122	100	150	400	630	376
512	160	131	240	520	880	497
768	257	179	770	880	1450	735
1024	400	250	1050	1700	2200	966

Language model: n-gram

# of streams	Throughput (RTFX)
32	2100

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 270

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	14	15	30	100	0.997
8	18.6	17	20	24	76	7.96
16	25	23	32	35	100	15.88
32	34	32	44	46	150	31.67
48	43	42	57	58.6	190	47.4
64	60	60	70	75	200	63
128	90	80	106	200	400	125.2
256	200	140	500	900	1000	246

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 660

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	22.6	44.6	46	100	0.997
64	80	70	160	200	200	63.1
128	120	96	250	300	400	125.3
256	180	154	400	500	700	247
384	240	190	500	700	900	364
512	330	250	700	1000	1300	476
768	800	400	2000	2000	3000	680
1024	2000	2000	5000	5000	5000	840

Language model: n-gram

# of streams	Throughput (RTFX)
32	320

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 366

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	10	11	11.7	30	1
8	13	12.3	16	20	38	7.99
16	16	15	22	23	41	15.97
32	23	23.7	32	33	50	31.9
48	28	28	38	41	50	47.9
64	32.5	35	45	46.5	60	63.8
128	46	45	63	65	75	127.5
256	72	73	103	106	140	254

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1620

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	11	20	30	80	1
64	38	43	50	60	80	63.8
128	50	49	67	80	95	127.6
256	75	78	106	110	135	254.6
384	99	100	146	150	178	381
512	119	123	183	188	209	507
768	165	170	265	275	290	758
1024	208	210	340	350	390	1006

Language model: n-gram

# of streams	Throughput (RTFX)
32	447

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 280

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	11.36	14	20	100	0.997
8	15.6	13.6	17	20	62	7.97
16	20	18	24	28	80	15.9
32	28	26	34	40	100	31.7
48	36	34	45	50	150	47.5
64	45	46	57	63	200	63
128	70	62	90	200	400	125
256	140	101	300	500	700	247

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	16	43	100	100	0.997
64	70	54	200	200	200	63
128	90	70	200	300	300	125.4
256	150	114	340	440	600	247
384	190	142	500	600	700	364
512	280	185	600	900	1200	476
768	560	294	1500	1600	2200	685
1024	1300	900	3000	3000	4000	860

Language model: n-gram

# of streams	Throughput (RTFX)
32	194

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 360

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	10.3	11.3	11.4	50	0.999
8	13	12.4	14	20	49	7.99
16	17.4	16	22	24.3	60	15.96
32	23	22	31	33	80	31.9
48	30	29	40	41.5	100	47.8
64	34	40	44	46	110	63.7
128	49.2	45	64	66	170	127
256	85	76	106	130	460	252

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1390

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	11.1	20	50	80	1
64	40	40	50	126	140	63.7
128	60	48	70	160	220	127
256	90	78	108	270	400	252
384	120	105	150	350	550	376
512	148	126	190	450	700	498
768	230	182	560	640	1200	738
1024	330	243	900	1160	1550	970

Language model: n-gram

# of streams	Throughput (RTFX)
32	451

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 295

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	10.4	20	30	100	0.995
8	15	12.4	29	30	39	7.97
16	20	16	40	42	50	15.92
32	25.7	23	46	58	64	31.7
48	31.2	29	53	68	79	47.5
64	37	36.6	60	75	100	63.2
128	60	50	110	134	210	125.5
256	150	97	380	400	505	242.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 800

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	11	100	100	100	0.996
64	60	46	130	140	140	62.8
128	90	60	200	207	211	124
256	142	100	330	377	386	241.7
384	190	130	450	510	560	354
512	242	160	600	680	730	460
768	430	370	900	1060	1300	660
1024	690	500	1260	1500	1800	840

Language model: n-gram

# of streams	Throughput (RTFX)
32	110.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 254

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	10.5	30	60	100	0.993
8	20	20	40	50	50	7.9
16	20.5	16	40	60	60	15.8
32	28	26	50	78	90	31.4
48	35	30	44	100	120	46.7
64	40	34	60	120	140	62
128	70	50	130	200	270	121.3
256	180	107	500	600	700	231.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 600

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	40	34	100	100	100	0.992
64	90	60	150	150	160	61.8
128	120	75	223	225	230	121.5
256	190	110	380	400	412	233
384	254	159	530	570	590	338
512	320	200	700	750	780	432
768	630	600	1100	1300	1400	605
1024	1370	1440	2140	2300	2500	672

Language model: n-gram

# of streams	Throughput (RTFX)
32	72.2

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 326

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10	10	12	20	80	0.998
8	14	12.2	20	20	47	7.97
16	19	15.6	24	28	60	15.9
32	24.7	24	33	37	86	31.75
48	32	30	42	44	110	47.5
64	35.6	37	45	50	124	63.3
128	53	46	66	100	230	125.5
256	102	79	180	340	460	246.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1020

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	11.3	50	80	80	0.998
64	50	43	120	130	137	63.2
128	74	52	170	210	220	125.5
256	115	84	300	360	390	247
384	150	112	400	500	550	365
512	190	140	500	630	730	479
768	320	220	700	950	1250	697
1024	500	300	1100	1360	1750	900

Language model: n-gram

# of streams	Throughput (RTFX)
32	180

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 333

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	10	11.3	13	40	1
8	13	12	16	20	40	7.99
16	17.3	14.7	22	27	60	15.96
32	25	25	33	37	80	31.9
48	29	28	38	42	90	47.8
64	34	34	45	47.6	110	63.7
128	53	45.5	66	120	220	127
256	110	80	228	380	540	252

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1230

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	10.6	20	40	80	1
64	45	40	100	104	126	63.7
128	70	49	160	188	220	127
256	115	82	280	343	386	252.7
384	156	110	400	500	560	376.5
512	193	134	510	620	700	499
768	360	240	740	1000	1300	740
1024	580	350	1200	1443	1740	976

Language model: n-gram

# of streams	Throughput (RTFX)
32	2200

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 330

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	10.8	12	20	80	0.997
8	14	12.3	14	20.4	52	7.97
16	18	15.2	22	26	65	15.92
32	26	25	34	37.4	80	31.76
48	30	30	40	44	110	47.5
64	37	40	47	50	132	63.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1070

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	14	50	80	80	0.997
64	53	45	130	134	140	63.3
128	77	55	200	220	230	125.6
256	118	86	300	370	400	247.4
384	158	116	410	500	580	365.4
512	195	140	520	650	740	480

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	193.6

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 293

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	10.4	14	30	100	0.997
8	16	13	25	36.5	53	7.97
16	20	15.6	35	50	64	15.92
32	27	25	40	64	90	31.77
48	34	30	52	80	100	47.5
64	40	40	60	103	126	63.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1070

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	15	35	100	100	0.997
64	53	45	120	134	140	63.3
128	78	55	170	220	226	125.5
256	116	85	300	360	390	247.4
384	157	116	386	500	570	366
512	195	142	500	632	740	481

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	194

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 25

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24	22.6	23.6	24	50	0.999
4	30.6	29.8	31	34	60	3.99
8	44.6	52	54	54.7	92.4	7.97

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 124

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	24.4	26	41	90	0.999
4	34.3	32.4	34	55	64	3.99
8	48.2	54	60	73	91	7.98

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	140
False	4	367
False	8	391

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 367

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	11	11.6	12	50	1
8	14	13	16	20	32.5	7.99
16	18	16	23	25	35	15.97
32	23	24	32	33	40	31.93
48	28	29	38	41	50	47.9
64	33	36	46	47.7	50	63.8
128	47	47	64	66	70	127.6
256	73	75	104	107	112	254.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1600

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	11.7	20	20	100	1
64	37	42	50	54	64	63.8
128	50.6	50	68	73	85	127.6
256	76	76	107	112	128	254.6
384	100	102	148	158	206	380.6
512	127	126	190	220	290	506
768	180	170	270	350	460	753
1024	230	220	360	485	640	996

Language model: n-gram

# of streams	Throughput (RTFX)
32	200

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 300

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12	13	14	40	1
8	15	14.3	20	20	33.4	7.99
16	20	20	27	30	37	15.97
32	27	27	36	40	44	31.93
48	34	36	47	50	56	47.9
64	41	44	56	58	65	63.8
128	64	67	83	87	100	127.5
256	91.4	95	126	130	143	254.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1300

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	13.5	20	23	100	1
64	45	50	60	70	73	63.8
128	70	70	90	100	110	127.5
256	100	100	150	160	200	254.4
384	128	130	191	200	230	380.6
512	160	160	240	250	300	506
768	220	220	350	370	460	753
1024	280	280	450	500	640	995

Language model: n-gram

# of streams	Throughput (RTFX)
32	700

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 290

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	12	15	20	100	0.997
8	16.4	14.3	20	20	61	7.96
16	21	20	27	30	70	15.9
32	29	27	37	42	100	31.76
48	36	35	48	53	130	47.5
64	45	50	59	63	160	63.2
128	80	70	100	200	300	126
256	120	100	200	300	500	248.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 900

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	18	50	100	100	0.997
64	60	54	100	200	200	63
128	90	76	200	200	300	126
256	130	110	300	350	500	248
384	170	146	360	500	600	367
512	200	180	400	600	800	485
768	400	280	800	1000	1000	710
1024	500	400	1000	2000	2000	924

Language model: n-gram

# of streams	Throughput (RTFX)
32	2300

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 358

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	11.4	12.2	12.5	20	1
8	15	15	20	20	30	7.99
16	18	16	23	25	40	15.98
32	24	24	32	34	52.4	31.95
48	30	30	40	43	60	47.9
64	35	40	47	48	70	63.9
128	51	47	65	67	180	127.3
256	87	78	105	133	500	253

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1380

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	12.1	13.6	27	100	1
64	42	44	52	90	128	63.7
128	60	50	72	150	220	127
256	90	80	110	240	380	252.7
384	120	106	153	320	570	377
512	147.7	130	195	410	710	500
768	230	183	600	630	1230	740
1024	356	243	840	1410	1800	972

Language model: n-gram

# of streams	Throughput (RTFX)
32	470

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 310

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	13.7	14.8	15	33	1
8	17	16	20	20	46	7.99
16	21	19.6	26	30	60	15.97
32	30	29.5	40	40	90	31.9
48	35	35	47	50	110	47.9
64	45	50	60	60	130	63.8
128	72	70	90	120	220	127.4
256	140	110	200	400	700	253.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1100

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22	20.6	22	40	100	0.999
64	70	60	100	100	300	63.7
128	100	80	200	300	600	126.6
256	200	120	400	700	1000	250
384	300	150	800	2000	2000	373
512	400	200	1000	2000	3000	490
768	1000	600	4000	5000	6000	730
1024	2000	2000	5000	6000	7000	960

Language model: n-gram

# of streams	Throughput (RTFX)
32	110

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 270

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	12.7	13.4	13.7	40	0.999
8	15.3	14.3	15.5	20	45.5	7.99
16	21	18	26	28	60	15.97
32	28	28	37	40	90	31.9
48	35	35	46	47.4	100	47.8
64	42	40	54	55	130	63.7

Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1240

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	13.7	20	30	100	1
64	53	50	64	120	150	63.7
128	77	65	96	200	245	127
256	120	107	156	300	440	252.5
384	162	145	220	440	640	376
512	200	180	276	530	800	499

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	300
False	32	2200
True	32	170

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 190

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	13.5	14	14.6	30	1
8	17.5	17	18	19	35	7.99
16	24	22	32.3	33	50	15.97
32	33.6	38	43	46	66	31.9
48	43	46	60	60.3	90	47.85
64	53	63	71	72	113	63.8

Chunk size (ms): 960
Language model: none
Maximum effective # of streams without language model (greedy generation): 997

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	13.5	20	30	100	1
64	60	70	77	80	90	63.8
128	87.2	82	122	123.7	140	127.4
256	140	140	215	217	230	254
384	192	194	310	314	330	380
512	250	250	400	410	510	503

Language model: none

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	400
False	32	2270
True	32	130

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	16.7	17.4	20	40	0.999
8	21	19.8	21	30	50.4	7.99
16	29	26	40	41	70	15.96
32	41	45	51	55	110	31.9
48	53	58	71	73	160	47.75
64	65.5	72	83	85	210	63.6

Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	18.4	20	36	100	0.999
64	80	83	95	140	200	63.7
128	119	100	155	230	296	126.8
256	190	170	270	400	530	252
384	260	245	378	580	800	374.5
512	346	330	496	850	1200	494

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	200
False	32	2000
True	32	120

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 178

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10000000	10000000	10000000	10000000	10000000	0.999
8	10000000	10000000	10000000	10000000	10000000	7.99
16	10000000	10000000	10000000	10000000	10000000	15.96
32	10000000	10000000	10000000	10000000	10000000	31.86
48	10000000	10000000	10000000	10000000	10000000	47.7
64	10000000	10000000	10000000	10000000	10000000	63.5

Chunk size (ms): 960
Language model: none
Maximum effective # of streams without language model (greedy generation): 810

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10000000	10000000	10000000	10000000	10000000	0.999
64	10000000	10000000	10000000	10000000	10000000	63.7
128	10000000	10000000	10000000	10000000	10000000	127
256	10000000	10000000	10000000	10000000	10000000	252
384	10000000	10000000	10000000	10000000	10000000	376
512	10000000	10000000	10000000	10000000	10000000	498

Language model: none

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	40
False	32	310

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 126
Maximum effective # of streams without language model (greedy generation): 127

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	17	16	17	17.7	40	0.999
n-gram	8	24.5	22.6	24.4	34	71	7.98
n-gram	16	33	30	40	42.4	110	15.94
n-gram	32	45	44	56	60	160	31.8
n-gram	48	62	66.6	74	100	260	47.6
n-gram	64	84	84	93	150	350	63.3
none	1	15.6	15	15.7	16	40	0.999
none	8	22.4	20.8	21.8	32	60	7.98
none	16	29	26	35	38	90	15.95
none	32	41.5	44	50	55	140	31.84
none	48	58.6	64	70	86	240	47.6
none	64	80	80.4	88	130	310	63.4

Chunk size (ms): 800
Maximum effective # of streams with n-gram language model: 565
Maximum effective # of streams without language model (greedy generation): 569

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	20	17.7	18.7	36.6	60	0.999
n-gram	64	87	90	100	200	270	63.5
n-gram	128	140	123	164	350	500	126
n-gram	256	236	209	300	500	1000	249
none	1	17	14.8	15.7	30	60	1
none	64	78.4	83	90	170	256	63.5
none	128	130	111	155	340	480	126
none	256	226	200	290	470	900	249
none	384	357	300	750	1070	1600	368

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 126

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	16	17.4	23	60	0.998
8	24.1	22.6	30	41.4	50	7.98
16	32.7	30.3	40	50	82	15.95
32	44	48	55	63	130	31.84
64	79	86	93	130	290	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 463

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	17	40	70	70	0.998
64	100	92	200	270	270	62.8
128	165	126	360	450	490	123.6
256	290	260	640	800	1100	240

Language model: n-gram

# of streams	Throughput (RTFX)
32	224

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 132

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	15.7	17	18	40	1
8	23.7	22	30	33	50	7.99
16	31.1	29.5	40	40	80	15.98
32	42	40	55	57	100	31.9
64	74	83.8	90	93	280	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 576

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	16.7	24	40	60	1
64	85	87	104	240	260	63.8
128	137	120	170	340	470	127.2
256	234	210	310	600	1000	253

Language model: n-gram

# of streams	Throughput (RTFX)
32	455

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 120

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	15.5	16.4	20	60	0.997
8	24.4	21.97	24	33	74	7.95
16	33	29.4	40	42	105	15.84
32	45	47	53	60	150	31.5
64	80	84	91	160	310	62.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 400

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	17	50	60	60	0.997
64	104	88	260	263	270	62.4
128	172	120	360	470	500	122
256	330	256	800	970	1170	233.3

Language model: n-gram

# of streams	Throughput (RTFX)
32	167

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 133

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16.6	16.2	17.4	18	30	0.999
8	23.3	22.8	26	31	40	7.99
16	31.6	30.6	40	41	50	15.97
32	45	50	58	59	65.8	31.9
64	72	86	92	93	104	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 596

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	17.8	20	30	70	0.999
64	79.5	92	105	109	114	63.7
128	126	124	165	200	220	127
256	215	210	301	380	500	251

Language model: n-gram

# of streams	Throughput (RTFX)
32	424

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 131

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	15.6	16.3	16.5	50	0.999
8	24	22.8	24	30	75	7.98
16	32	30.2	30	40	109	15.95
32	43	40	54	55	140	31.83
48	57	66.8	70	74	150	47.7
64	74	84	90	92	200	63.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 540

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	16.7	17.7	33	60	0.999
64	85	88	100	200	265	63.5
128	140	121	160	380	550	125.8
256	262	209	550	720	1270	247

Language model: n-gram

# of streams	Throughput (RTFX)
32	990

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 108

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22	20.4	22.1	23	90	0.996
8	32.5	30	33.5	40	120	7.94
16	44	43	51	54	170	15.83
32	70	70	80	100	250	31.5
48	82	80	95	160	330	47.1
64	106	110	120	200	460	62.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 380

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	35	32	60	60	90	0.996
64	140	120	300	350	400	62.5
128	220	200	470	600	700	123
256	400	330	800	1100	1500	238

Language model: n-gram

# of streams	Throughput (RTFX)
32	875

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 135

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	15.9	16.6	17	30	0.999
8	24	22.6	30	32	50	7.98
16	30.7	29.8	36	40.6	57.6	15.97
32	42	48	55	56	70	31.9
64	70	82	88	90	140	63.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 624

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	17.2	20	30	60	0.999
64	75	88	100	100	120	63.7
128	120	120	160	165	188	127
256	194	200	288	292	320	253

Language model: n-gram

# of streams	Throughput (RTFX)
32	437

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 114

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18.8	17	20	20.6	90	0.996
8	28	25.2	30	36	100	7.94
16	39	34.8	40	50	150	15.84
32	53	50	63	78	210	31.53
48	71	80	86	130	280	47.1
64	96.5	99	112	230	400	62.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 410

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	23	60	90	90	0.996
64	120	108	200	300	300	62.6
128	200	160	440	500	600	123
256	360	300	700	1000	1300	238

Language model: n-gram

# of streams	Throughput (RTFX)
32	250

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 131

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	16.2	17	17.2	50	0.999
8	23.5	22.5	23.5	24	70	7.98
16	32	30	39	40	100	15.94
32	43	40	55	57	130	31.8
64	74	84.3	90	92	200	63.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 544

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	17.1	20	50	60	0.999
64	86	88	100	200	270	63.4
128	142	120	160	360	520	125.7
256	240	205	350	650	1000	247.5

Language model: n-gram

# of streams	Throughput (RTFX)
32	440

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 112

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	16	26	40	70	0.995
8	26	22.6	40	52	53	7.95
16	36	30.34	56	82	84	15.87
32	48	49	80	93.6	130	31.54
64	100	88	200	216	330	62

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 357

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	17	70	70	70	0.995
64	123	92	260	260	262	61.6
128	205	140	460	480	480	119.3
256	380	300	770	900	1000	225.5

Language model: n-gram

# of streams	Throughput (RTFX)
32	109.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 97

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	16.2	36	40	80	0.992
8	28	23.4	60	70	76	7.88
16	38	31	90	100	105	15.66
32	52	50	60	153	160	30.8
64	125	93	300	400	440	59.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 290

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	35	70	70	70	0.992
64	160	140	276	280	280	60.2
128	264	176	490	500	500	114.8
256	484	390	870	1000	1100	211.6

Language model: n-gram

# of streams	Throughput (RTFX)
32	71.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 124

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	16	18	20	60	0.997
8	24.7	22.4	27	32	74	7.95
16	33.3	29.9	39	40	105	15.86
32	46	40	55	60	150	31.55
64	80	84.5	92.5	150	305	62.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 430

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	17	50	60	60	0.997
64	105	90	254	270	300	62.5
128	170	125	360	470	500	122.6
256	302	250	640	810	1030	236

Language model: n-gram

# of streams	Throughput (RTFX)
32	178

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 127

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	15.3	16.4	20	40	0.999
8	23.3	21.7	24	33	70	7.98
16	30	28	37	40	95	15.95
32	43	40	52	57	140	31.8
48	59	64.5	70	105	240	47.6
64	81	81.7	91	140	400	63.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 480

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	16.3	30	40	60	0.999
64	103	86.3	243	250	265	63.5
128	174	130	400	460	490	126
256	312	250	650	800	1040	248.4

Language model: n-gram

# of streams	Throughput (RTFX)
32	960

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 123

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	16	17.3	20	70	0.997
8	24.7	22.5	24.7	35	79	7.95
16	33.7	30	40	42	114	15.86
32	45	40	56	58	160	31.56
48	60.4	66.4	71	95	230	47.1

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 453

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	20	60	70	70	0.996
64	110	94	263	280	300	62.6
128	180	130	400	500	510	122.6
256	320	260	680	900	1100	236.8

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	190

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 112

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	16	21.5	35	80	0.997
8	28	23	50.3	68	81	7.95
16	38	30.3	77	96	110	15.86
32	50.6	48	90	130	160	31.6
48	72	67.6	130	190	260	47.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 460

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	21	50	80	80	0.996
64	110	93	270	280	280	62.5
128	175	126.5	400	500	500	123
256	310	260	700	800	1100	237.3

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	170

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 10

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	38.5	37.7	38.6	39.3	73	0.998
4	69	67	70	80	140	3.98
8	109.8	126	131	150	240	7.93

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 49

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	41	40	40	57	80	0.998
4	72	69.6	80	100	108	3.99
8	109	126	132	140	178	7.96

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	84
False	4	146
False	8	148

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 133

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	17	18	18.3	30	0.999
8	23.44	23.2	24.56	29	40	7.99
16	31.8	31	40	41	50	15.97
32	44	40	58	59	66	31.9
48	56	67	71.5	73	83	47.8
64	74	87	93	95	107	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 590

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	18	23	30	80	0.999
64	79	90	105	110	115	63.7
128	128	124	167	184	224	127
256	218	214	310	380	500	251

Language model: n-gram

# of streams	Throughput (RTFX)
32	140

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 120

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	17.8	19	19.3	30	0.999
8	26.7	26.4	28	30	45	7.99
16	38	37	40	46.3	55.7	15.97
32	50	50	64	66	75	31.9
48	67	79	85	87	98	47.8
64	87.4	103.7	110	110	130	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 550

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	20	25	30	70	0.999
64	94	110	120	130	140	63.7
128	147	147	190	200	220	127
256	240	245	350	360	420	252

Language model: n-gram

# of streams	Throughput (RTFX)
32	1450

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 115

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	18	21	22	80	0.996
8	29	26	30	38	100	7.95
16	39.5	37	43	47	114	15.88
32	56.6	60	70	75	170	31.6
48	70	78	87	90	200	47.3
64	90	103	114	120	220	63

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 444

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	26	65	90	90	0.995
64	120	110	200	300	300	63
128	180	160	300	380	500	124
256	280	270	400	600	700	244

Language model: n-gram

# of streams	Throughput (RTFX)
32	1450

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 130

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	17.14	18	18.24	27	1
8	24.1	23.6	25	30	54	7.99
16	32	31	40	42	90	15.98
32	44	47	60	60	100	31.9
64	77.7	86	92	95	280	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 543

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	18.7	20	35	70	0.999
64	87	92	100	181	264	63.5
128	142	125	170	340	504	126
256	250	200	400	660	1180	248

Language model: n-gram

# of streams	Throughput (RTFX)
32	500

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 120

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	20	21.2	21.6	49.2	0.999
8	29.5	29	31	38	70	7.99
16	41.1	40.3	46	50	96	15.97
32	54	53	66	70	158	31.9
64	93	106	114	120	220	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 460

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	28	30	50	100	0.998
64	130	123.5	200	200	500	63.4
128	200	170	300	400	1000	125
256	500	300	1000	2000	3000	248

Language model: n-gram

# of streams	Throughput (RTFX)
32	82

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 97

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23	22	23	24	50	0.999
8	30	29	30	46	80	7.98
16	39.6	37	50	54	110	15.94
32	57.3	65	70.4	73	210	31.8
48	80	90	93	106	300	47.6

Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 444

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	28	40	50	80	0.999
64	126	133	150	230	295	63.4
128	204	180	250	420	570	125.8
256	360	364	510	840	1300	247

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	200
False	32	1140
True	32	95

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 53

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23	22.6	23.24	23.3	40	0.999
8	41	40.3	41.5	45	71	7.98
16	56	55	66	70	100	15.94
32	86	105	109	110	200	31.8
48	132	150	158	195	300	47.6
64	2670	2520	4890	5090	6130	55.1

Chunk size (ms): 960
Language model: none
Maximum effective # of streams without language model (greedy generation): 280

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25	23.2	30	40	70	0.999
64	180	213	220	230	237	63.5
128	306	280	410	415	485	126
256	574	640	900	1090	1280	246
384	5700	5280	10300	10950	12600	282.3

Language model: none

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	200
False	32	1400
True	32	93

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 56

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	36	35.5	37	40	68	0.998
8	46	44	50	70	100	7.97
16	63.4	58.5	85	88	155	15.92
32	97	110	123	140	230	31.7
48	146	150	200	260	400	47.4

Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 272

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	40	39	50	60	90	0.998
64	200	220	240	400	420	63.2
128	330	280	416	610	770	125
256	690	640	1350	1500	2300	242.7

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	140
False	32	730
True	32	80

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 50

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10000000	10000000	10000000	10000000	10000000	0.997
8	10000000	10000000	10000000	10000000	10000000	7.97
16	10000000	10000000	10000000	10000000	10000000	15.9
32	10000000	10000000	10000000	10000000	10000000	31.6
48	10000000	10000000	10000000	10000000	10000000	47.1
64	10000000	10000000	10000000	10000000	10000000	51.1

Chunk size (ms): 960
Language model: none
Maximum effective # of streams without language model (greedy generation): 248

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10000000	10000000	10000000	10000000	10000000	0.998
64	10000000	10000000	10000000	10000000	10000000	63.2
128	10000000	10000000	10000000	10000000	10000000	124.8
256	10000000	10000000	10000000	10000000	10000000	238
384	10000000	10000000	10000000	10000000	10000000	245

Language model: none

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	17
False	32	105.4

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 48
Maximum effective # of streams without language model (greedy generation): 48

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	25.5	24.55	25.7	26	50	0.998
n-gram	8	49.2	48	50.7	64	140	7.97
n-gram	16	54	50	60	70	160	15.9
n-gram	32	88	80	101	160	280	31.7
n-gram	64	5000	4700	9600	10300	12300	48.7
none	1	23.4	22.7	23.3	24	50	0.999
none	8	47	46.6	48.5	60	130	7.97
none	16	51	49	56	60	150	15.92
none	32	87	80	100	164	292	31.7
none	64	5400	5000	10200	11000	13000	48

Chunk size (ms): 800
Maximum effective # of streams with n-gram language model: 227
Maximum effective # of streams without language model (greedy generation): 226

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	28	25.5	26.4	46.6	70	0.999
n-gram	64	170	193	200	340	420	63.2
n-gram	128	310	303	395.5	600	800	125
n-gram	256	2000	1800	3300	4000	4500	227
none	1	26	23.66	24	40	60	0.999
none	64	162	188	194	320	410	63.2
none	128	306	290	390	610	800	125
none	256	2000	1900	3400	4000	5000	226

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 48

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24	23.53	24.3	33	60	0.998
8	50	49.5	53	60	101	7.97
16	55	52	60	70	130	15.93
32	84	80	100	120	210	31.7
64	2200	2000	4000	4700	5100	49

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 210

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	25	44	70	70	0.997
64	185	190	330	396	400	62.2
128	340	300	630	770	790	121
256	1860	2000	3200	3800	4500	207

Language model: n-gram

# of streams	Throughput (RTFX)
32	223.6

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 49

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23.5	23	24	25.3	40	0.999
8	50	49.1	51	60	103	7.99
16	53	53	60	64	120	15.97
32	86	85	99	102	200	31.9
64	4400	4000	8600	9000	11000	49.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 228

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	27	25.36	34	40	60	0.999
64	170	190	210	300	400	63.5
128	310	300	396	640	777	126.3
256	2200	2000	4084	4500	5200	224

Language model: n-gram

# of streams	Throughput (RTFX)
32	410

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 48

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25	23.5	24.6	30	60	0.996
8	49	48	51	60	125	7.91
16	52	50	60	70	150	15.8
32	90	90	98	160	300	31.3
64	1600	1500	2800	3000	4400	49

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 180

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	24.5	60	60	60	0.996
64	200	190	400	400	420	61.5
128	430	380	900	1000	1200	117
256	2500	2700	4700	4700	6300	177

Language model: n-gram

# of streams	Throughput (RTFX)
32	166.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 49

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24.5	24	25	26	40	0.999
8	50	49.9	51.5	60	69	7.98
16

NVIDIA Riva

Performance

Contents

Performance#

Evaluation Process#

Results#