Performance#

Evaluation Process#

This section shows the latency and throughput numbers for streaming and offline configurations of the Riva ASR service on different GPUs. These numbers were captured after the preconfigured ASR pipelines from our Quick Start scripts were deployed. The Jasper, QuartzNet, Conformer, and Citrinet-1024 acoustic models were tested.

In streaming mode, the client and the server used audio chunks of the same duration (100ms, 160ms, and 800ms depending on the server configuration). Refer to the Results section for the chunk size value to use.

The Riva streaming client riva_streaming_asr_client, provided in the Riva image, was used with the --simulate_realtime flag to simulate transcription from a microphone, where each stream was doing three iterations over a sample audio file (1272-135031-0000.wav) from the LibriSpeech dev-clean dataset. The Librispeech datasets can be obtained from https://www.openslr.org/12.

The source code for the riva_streaming_asr_client can be obtained from https://github.com/nvidia-riva/cpp-clients.

The command used to measure performance was:

riva_streaming_asr_client \
   --chunk_duration_ms=<chunk_duration> \
   --simulate_realtime=true \
   --automatic_punctuation=true \
   --num_parallel_requests=<num_streams> \
   --word_time_offsets=true \
   --print_transcripts=false \
   --interim_results=false \
   --num_iterations=<3*num_streams> \
   --audio_file=1272-135031-0000.wav \
   --output_filename=/tmp/output.json

The riva_streaming_asr_client returns the following latency measurements:

intermediate latency: latency of responses returned with is_final == false
final latency: latency of responses returned with is_final == true
latency: the overall latency of all returned responses. This is what is tabulated in the following tables.

Refer to the following diagram for a schematic representation of the different latencies measured by the Riva streaming ASR client.

Schematic Diagram of Latencies Measured by Riva Streaming ASR Client

In offline mode, the command used to measure maximum throughput was:

riva_asr_client \
   --automatic_punctuation=true \
   --num_parallel_requests=32 \
   --word_time_offsets=true \
   --print_transcripts=false \
   --num_iterations=96 \
   --audio_file=1272-135031-0000x5.wav \
   --output_filename=/tmp/output.json

where 1272-135031-0000x5.wav is simply the 1272-135031-0000.wav audio file concatenated five times. The source code for the riva_asr_client can be obtained from: https://github.com/nvidia-riva/cpp-clients

Results#

Latencies and throughput measurements for streaming and offline configurations are reported in the following tables. Throughput is measured in RTFX (duration of audio transcribed / computation time).

Note

Audio files were iterated 1 time for Xavier AGX, Xavier NX, and Orin AGX and 3 times for all other experiments.

Note

If the language model is none, the inference is performed with a greedy decoder. If the language model is n-gram, then a beam decoder was used.

Note

The values in the tables are average values over 3 trials. The values in the table are rounded to the last significant digit according to standard deviation calculated on 3 trials. If a standard deviation is less than 0.001 of the average, then the corresponding value is rounded as if standard deviation equals 0.001 of the value.

For specifications of the hardware on which these measurements were collected, refer to the Hardware Specifications section. Please notice, that

results on AWS and GCP are computed using Riva 2.4.0
results On-Prem on GPUs A40, T4, L40, H100 on Riva 2.9.0.
other On-Prem results on Riva 2.8.0

Cloud instance descriptions for AWS and GCP.

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 180
Maximum effective # of streams without language model (greedy generation): 192

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	13.9	13.57	13.9	14.2	28	1
n-gram	8	26.2	25.5	26	27	60	7.98
n-gram	16	37	35	45	45.4	90	15.93
n-gram	32	52	56	63	64	140	31.8
n-gram	48	62	65	70	72	150	47.6
n-gram	64	76	76	81	83	230	63.3
none	1	13.1	12.8	12.96	13.2	22	1
none	8	25	24.5	25	25	60	7.98
none	16	34	30	44	44	80	15.94
none	32	50	50	61	63	120	31.8
none	48	56	62	66	70	140	47.6
none	64	70	73	76	77	190	63.3

Chunk size (ms): 800
Maximum effective # of streams with n-gram language model: 690
Maximum effective # of streams without language model (greedy generation): 750

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	16	15.4	16.3	24	30	1
n-gram	64	97	96	107	170	300	63.3
n-gram	128	140	130	168	270	530	125.7
n-gram	256	230	220	270	530	1170	247
n-gram	384	330	330	380	740	2000	365
n-gram	512	470	440	500	1240	3060	478
none	1	13	12.67	13	17.1	18	1
none	64	86	88	93	160	290	63.4
none	128	120	100	140	250	500	125.9
none	256	200	200	240	470	1090	247.7
none	384	300	304	340	690	1700	365.6
none	512	420	410	460	1040	2870	480

Speaker Diarization	Language model	# of streams	Throughput (RTFX)
False	n-gram	32	1500
False	none	32	1600
True	n-gram	32	120
True	none	32	120

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 150

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.8	13.7	14.3	14.6	20	1
8	26.6	26.3	27.7	30	40	7.98
16	37	36	46	47	50	15.95
32	57	63	70	70	73	31.84
48	67	73	80	80	90	47.7
64	80	87	94	95	100	63.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 740

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14.5	15.6	20	20	0.999
64	95	103	111	114	117	63.4
128	150	150	200	200	200	126.2
256	220	235	270	300	300	250.5
384	300	340	400	410	440	372
512	400	470	540	560	650	492
768	660	730	1000	1000	1200	720

Language model: n-gram

# of streams	Throughput (RTFX)
32	1500

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 182

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.5	13.3	13.8	14	20	1
8	25.7	25	25.7	27	52	8
16	40	30	44	45	80	15.97
32	50	50	62	63	127	31.95
48	61.6	65	70	74	183	47.9
64	74.6	75	81	87.7	270	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 704

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14	20	21	20	1
64	93	94	110	175	300	63.8
128	140	130	160	300	530	127.4
256	228	210	270	585	1190	253.5
384	390	320	780	1300	2100	371
512	620	430	1720	2440	3400	483
768	1500	660	4400	7300	10000	682

Language model: n-gram

# of streams	Throughput (RTFX)
32	1000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.9	13.9	14.3	14.4	17	1
8	27.5	27.3	28.3	30	40	7.99
16	40	37	40	50	50	15.98
32	57	64	70	71	74	31.9
48	70	76	81.4	83	90	47.9
64	86.6	93	99.4	101	110	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 760

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14.8	16	17	20	1
64	90	100	110	115	140	63.7
128	140	135	180	200	270	127
256	230	230	330	400	430	252
384	340	340	646	700	800	374
512	460	470	800	970	1200	494
768	920	740	1900	2300	3000	727

Language model: n-gram

# of streams	Throughput (RTFX)
32	1900

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 188

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.5	13.37	13.7	13.8	20	1
8	27	25	30	30	30	7.99
16	34	30	44	44.5	50	15.97
32	50.3	57	62	63	69	31.9
48	59	65	68	70	80	47.9
64	70	76	80	81	90	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 800

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14.3	15	16	20	1
64	85	93	103	108	120	63.8
128	130	120	160	200	230	127.2
256	197	200	260	400	500	252.3
384	285	310	400	650	670	375.5
512	390	420	600	860	1000	496

Language model: n-gram

# of streams	Throughput (RTFX)
32	2800

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 90

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.1	14.9	15.4	16	20	0.999
8	34	34	40	40	56	7.97
16	48	48	60	60	90	15.9
32	73	77	90	90	150	31.6
48	92	100	106	110	200	47.25
64	110	115	125	140	270	62.8

Chunk size (ms): 800
Language model: none
Maximum effective # of streams without language model (greedy generation): 380

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18.3	17.6	20	20	20	0.999
64	130	124	160	240	300	62.7
128	220	200	300	500	700	123
256	400	370	700	1000	1300	237
384	680	556	1350	2000	2600	337
512	2000	2000	4000	6000	8000	400

Language model: none

# of streams	Throughput (RTFX)
32	870

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 190

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.6	13.48	14	14.5	20	1
8	25.5	25.4	26.2	26.7	34	7.99
16	35	32	44.4	45	48	15.97
32	50	56	63	63	70	31.9
48	58	64	69	70	97	47.8
64	70	74	79	82	118	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 860

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14.5	15	16	20	1
64	85	92	100	100	110	63.7
128	110	110	140	150	170	127.2
256	172	184	237	243	255	253.4
384	246	295	341	350	380	378.6
512	340	410	460	494	540	502
768	540	608	800	900	1150	740

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 175

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.5	13.13	13.7	17	20	1
8	26	24.7	26	32	57	7.98
16	34	30	44	46	94.5	15.93
32	50	50	62	80	160	31.8
48	63	63	70	82	243	47.6
64	78	75	83	128	300	63.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 510

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14	20	20	20	1
64	108	86	230	284	323	63.3
128	170	120	400	500	570	125.6
256	310	223	720	1080	1250	247
384	530	340	1350	1640	2150	364.4
512	1200	600	3000	3600	4300	468

Language model: n-gram

# of streams	Throughput (RTFX)
32	1000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 250

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.9	9.7	10	10.15	16.7	1
8	14.5	14	15	15.5	34	7.99
16	23	23	25	25.8	60	15.95
32	35	34	36	37	100	31.8
48	46	44	50	50	140	47.6
64	55	53	58	60	200	63.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 980

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	10.2	11	16	20	1
64	64	56	60	135	260	63.4
128	108	94	110	240	470	126
256	172	148	174	400	960	248.3
384	240	207	250	585	1430	367
512	320	270	320	800	2100	482
768	484	400	480	1400	3900	703

Language model: n-gram

# of streams	Throughput (RTFX)
32	4000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 260

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.53	9.53	9.8	9.9	13	1
8	13.7	13.6	14.4	15	19	8
16	20	20	24	25	27	16
32	34.5	34.6	36	37	41	31.96
48	43	43	47	48	51	47.9
64	54	54	60	60	68	63.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1240

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.3	10.3	11	11.5	20	1
64	55	55	62	65	70	63.9
128	96	96	108	112	120	127.5
256	150	160	190	200	220	254.4
384	190	203	240	240	260	381
512	240	265	306	316	340	506
768	350	397	460	480	520	755

Language model: n-gram

# of streams	Throughput (RTFX)
32	1000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 230

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10	9.9	10.3	11	14	1
8	16	16	17	20	23	8
16	24	25	30	29.5	32	15.98
32	42.5	42.4	44.5	46	50	31.94
48	53	53	60	63	66	47.9
64	60	60	66	70	74	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1000

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	10.7	11.4	12	20	1
64	69	70	80	80	86	63.8
128	100	104	140	150	200	127.2
256	176	184	260	290	330	253
384	260	260	400	460	600	377
512	310	334	500	610	645	499
768	500	500	800	1060	1400	737

Language model: n-gram

# of streams	Throughput (RTFX)
32	1000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 190

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.2	11.07	11.3	11.4	15	1
8	20	20	20.8	21	30	8
16	30	30	37	38	45	15.98
32	52	53	55	56	64	31.9
48	64	64	70	71.7	75	47.9
64	70	72	78	79	85	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 830

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.4	12.2	13	13.5	20	1
64	88	90	97	100	110	63.8
128	140	130	170	200	240	127
256	210	220	270	423	460	252
384	290	314	380	620	670	375.6
512	382	410	530	900	960	496
768	670	626	1350	1700	2200	733

Language model: n-gram

# of streams	Throughput (RTFX)
32	1000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 150

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.2	11.7	17	29	35.6	1
8	26	23	30	56	80	8
16	38	37	45	70	94	15.97
32	67	65	74	90	114	31.9
48	87	85	100.8	120	160	47.8
64	100	97	110	130	180	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 600

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20.5	17.7	34	35	40	1
64	116	110	160	170	200	63.8
128	190	180	260	270	300	127
256	300	310	400	420	600	252.5
384	460	470	700	750	900	374
512	600	610	900	1100	1200	492
768	1300	1300	2000	2300	3000	725

Language model: n-gram

# of streams	Throughput (RTFX)
32	600

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 180

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.1	11.05	11.46	11.85	20	1
8	19.2	19	20.1	20.4	25	8
16	30	30	38	38	43	15.97
32	50	50	55	57	70	31.9
48	63	61.6	70	73	90	47.85
64	70	70	78	80	100	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 850

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	11.9	12.5	12.7	20	1
64	86	88	97	100	105	63.8
128	120	120	150	150	160	127.3
256	184	210	236	240	260	253.4
384	255	300	340	344	356	379
512	343	400	460	510	600	502
768	570	620	900	1000	1400	740

Language model: n-gram

# of streams	Throughput (RTFX)
32	900

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 175

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.14	12.85	13.8	15	21	1
8	15.2	14.6	16.4	18.1	37	7.99
16	20	20	25	27	59	15.95
32	34	33.5	39	42	98	31.84
48	43	41	48	52	165	47.7
64	50	46	56	60	235	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 820

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17.5	17	22	25	27	1
64	72	57	130	210	280	63.4
128	123	96	230	380	500	126
256	200	140	400	700	1040	248
384	279	186	640	1140	1500	367.4
512	390	240	1110	1500	2180	483

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 290

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	7.1	6.8	8	9	16	1
8	10	9.2	11	12	37	7.99
16	14	13	16	18.5	60	15.96
32	18	16	20	24	100	31.85
48	23	19	26	30	154	47.7
64	28	23	30	37	196	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1000

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	11.75	16.5	17	20	1
64	55	40	110	197	263	63.4
128	85	56	190	350	470	126
256	146	86	350	656	940	248.6
384	207	110	550	993	1360	368
512	280	146	840	1320	1900	484
768	500	200	1600	2330	3200	700

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 98
Maximum effective # of streams without language model (greedy generation): 110

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	12.8	12.6	13	13	24.8	1
n-gram	8	26	25.3	26.2	28	53	7.98
n-gram	16	37	35	43	44	87	15.94
n-gram	32	60	64	68.3	70	160	31.8
n-gram	48	92.5	96	99.4	101	270	47.5
none	1	11.9	11.7	11.9	12	18.2	1
none	8	26	25.3	30	30	46	7.98
none	16	34	32	40	41	74	15.94
none	32	55.4	61	65	66	140	31.8
none	48	84	90	93	94.2	240	47.6
none	64	107	111	115	117	320	63.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 450

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14.7	15.5	22	23	1
64	130	126	140	200	320	63.3
128	203	200	225	330	560	125.6
256	354	360	420	640	1400	246.6

Language model: n-gram

Speaker Diarization	Language model	# of streams	Throughput (RTFX)
False	n-gram	32	1200
False	none	32	1200
True	n-gram	32	100
True	none	32	100

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 93

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	13	13.6	14	16	0.999
8	27	26	30	30	34	7.98
16	36	36	43	44	46	15.95
32	59	65	72	73	76	31.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 75

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.7	13.4	13.6	20.7	1
8	30	30	33	34	46	7.99
16	42	40	50	51	66	15.97
32	74	81	84	86	118	31.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 104

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.2	12	12.7	13	15	0.998
8	25.1	24.8	26	30	33	7.97
16	33	32	40.4	41.4	43	15.94
32	57.4	64	67	68	70	31.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 98

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.4	13.5	14	14.1	16	1
8	27	26.5	28.4	30	33	7.99
16	36	35.6	40	44	46	15.97
32	60	67	74	75	80	31.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 100

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.1	12.1	12.3	12.5	15	1
8	26	25.8	30	30	30	7.99
16	36	34.4	42.5	43	46	15.97
32	60	67	70	71	80	31.9
48	88	96	100	101	110	47.8

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 70

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.8	14.6	15	15.2	20	0.999
8	34	35	38	40	50	7.97
16	50	50	60	60	80	15.9
32	80	87	100	100	140	31.6
48	120	130	140	150	250	47.2

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 104

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.2	12.2	12.7	13	15	1
8	27	27	30	30	32	7.99
16	34	33	41	42	43	15.97
32	56	62	66	67	74	31.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 40

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19.2	16.5	27	35.3	45	0.997
8	45	42	60	70	95	7.95
16	76	75	90	100	126	15.85
32	130	130	155	170	190	31.55
48	3000	3000	5000	6000	7000	40

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 176

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	32	35.6	62	62	62	0.997
64	293	290	380	400	400	62.4
128	600	530	800	900	1000	120
256	5000	5000	7000	8000	9000	200

Language model: n-gram

# of streams	Throughput (RTFX)
32	650

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 110

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.1	12	12.6	12.8	15.6	1
8	25.7	25	30	30	32	7.99
16	35	35	41	41.3	45	15.97
32	54	60	64	65	70	31.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 480

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.9	14	15	16	1
64	110	120	134	136	150	63.7
128	190	200	220	230	250	126.8
256	330	370	410	420	440	252

Language model: n-gram

# of streams	Throughput (RTFX)
32	1400

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 66

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.63	13.9	17.5	19.6	19.84	0.997
8	32	30	38	43	45	7.94
16	47.8	48.8	60	80	83	15.8
32	87	85	160	170	172	31.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 330

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14	18	18	18	0.997
64	150	160	179	190	190	62.3
128	260	280	320	330	330	122
256	470	520	580	600	600	235

Language model: n-gram

# of streams	Throughput (RTFX)
32	1000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 70

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	13.7	16.7	17	18	0.996
8	29	28.8	32	34	37	7.94
16	42	40	50	54	60	15.8
32	80	80	130	140	160	31

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 98

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.2	12	12.7	14	15	0.998
8	25.3	25	26	28	30	7.98
16	35.6	34.4	41.8	42.5	46	15.93
32	56	63	67	68	74	31.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 99

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.1	11.9	12.4	15	18	1
8	26	25	26	31	52	7.98
16	35.7	33	41.6	44	82	15.94
32	56	60	64.4	80	154	31.8
48	90	90	95	123	297	47.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 12

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	28	27.3	28	33.5	52	0.998
4	53	51	52	62	108	3.986
8	104	100	102	124	240	7.94

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 62

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	29.7	29.2	30	37	40	0.999
4	55	53.2	60	70	75	3.99
8	104	101.6	110	125	140	7.97

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	142
False	4	190
False	8	195

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 180

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10	9.8	10.1	10.3	15.6	1
8	17.44	17	18.5	19	37	7.99
16	28	27	30	30	60	15.95
32	42	40	44	46	108	31.8
48	53	50	60	60	150	47.6
64	69	68	73	75	210	63.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 180

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.84	9.9	10.22	10.29	12	1
8	16	16	17	18	20	8
16	26.2	26.5	29	29	30	15.98
32	40	40	45	47	50	31.95
48	52	52	56	58	60	47.9
64	63	63	68.6	70	74	63.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 108

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.6	10.6	11	11.4	13	0.999
8	22	22	23	24	26	7.98
16	36	40	40	40	42	15.95
32	60	60	62	63	66	31.8
48	82	82	85.3	86.6	100	47.6
64	103	101	110	120	180	63.2

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 140

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.3	11.3	11.7	11.8	14	1
8	20.4	20	22	23	25	8
16	31.6	32.7	34	35	37	15.98
32	50	50	53	54.5	57	31.94
48	66	70	78	80	83	47.9
64	81	90	95	97	100	63.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 121

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.6	10.6	10.8	10.9	12.8	1
8	20.5	20.3	21	22	25	8
16	30	30	36	36	37	15.98
32	55.5	56.1	57.5	58	62	31.9
48	76	78	80.6	82	86.7	47.85
64	97	98	102	104	120	63.7

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 110

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	12.7	18	30	37	1
8	30	26	35	50	75	7.99
16	38	36	50	64	89	15.97
32	65	62	80	90	120	31.9
48	79	77	90	104	144	47.85
64	100	100	120	140	170	63.7

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 110

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	11	11.2	11.4	12	1
8	23	22.6	25	25	27	7.99
16	35	35	39	40	41	15.98
32	59	59	61	62	72	31.9
48	83	83	90	90	130	47.8
64	104	104	110	110	170	63.6

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 95

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	10.8	11.5	12	13	0.997
8	22	21	24	25	26	7.95
16	40	40	50	50	60	15.83
32	74	72	90	100	100	31.3
48	105	98	147	150	160	46.4
64	141	126	210	250	280	61.1

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 111

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.1	11	11.7	12	14	0.999
8	23	22.5	25	26	29	7.98
16	35	35	39	39.6	44	15.93
32	58	59	61	63	67	31.8
48	80	81	84	86	90	47.6
64	100	100	105	106	120	63.3

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 103

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	10.7	11.22	15.6	16.4	1
8	24	23	25	30	48	7.99
16	36	34	38.5	50	80	15.95
32	61	57	60.4	90	150	31.8
48	87	79.7	86	130	278	47.55
64	115	100.3	122.4	264	447	63.2

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 100

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.6	13.6	14.6	21	1
8	20	20	22	24	40	7.99
16	27	27	30	32	56	15.96
32	46	45	50	53	96	31.85
48	61	58	64	67	180	47.7
64	78	72	80	84	330	63.5

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 258

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	7.6	7.2	8.3	9	17	1
8	11	10	12	13	36	7.99
16	14	13	15	17	50	15.96
32	20	18	22	24	90	31.9
48	24	22	27	30	130	47.7
64	32	28	35	38	170	63.5

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 100
Maximum effective # of streams without language model (greedy generation): 111

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	18.93	18.52	18.83	19	39	0.999
n-gram	8	40	37	50	52.8	87	7.97
n-gram	16	53	50	66	67	130	15.9
n-gram	32	66	69	72.3	74	170	31.7
n-gram	48	91	94	100	103.6	280	47.4
none	1	18	17.6	17.7	17.85	33.6	0.999
none	8	38	36	40	40	82	7.97
none	16	48	44	62	64	120	15.9
none	32	60.5	66	69.4	70	160	31.73
none	48	84.7	89	93	96	280	47.4
none	64	108	109	117	122	390	63

Chunk size (ms): 800
Maximum effective # of streams with n-gram language model: 424
Maximum effective # of streams without language model (greedy generation): 455

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	21	20.4	21	31.3	32	0.999
n-gram	64	140	130	170	220	400	63.1
n-gram	128	210	200	230	390	740	124.8
n-gram	256	374	360	420	770	2100	243
none	1	18.2	17.6	18	25	26	1
none	64	120	110	150	230	400	63.1
none	128	190	190	210	378	720	124.9
none	256	336	310	390	740	2060	244
none	384	550	500	580	1490	4640	357

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 97

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19.13	19.14	19.7	19.9	22	0.999
8	40	40	40	50	53	7.97
16	54	53	67	70	72	15.93
32	66	73.4	80	80	83	31.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 460

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	20	21	22	22	0.998
64	120	120	140	150	160	63.3
128	200	210	230	240	250	125.5
256	340	370	410	420	500	248

Language model: n-gram

# of streams	Throughput (RTFX)
32	1000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 103

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	17.8	18.3	18.5	27	1
8	37	35.8	37.3	45	80	7.99
16	50	50	64	66	109	15.97
32	64	68	71	80	180	31.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 450

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19.5	19	20	28.7	29	1
64	130	120	166	240	400	63.7
128	200	195	220	400	730	127
256	434	350	850	1500	2200	247.6

Language model: n-gram

# of streams	Throughput (RTFX)
32	640

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 94

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19.1	19.1	19.6	19.7	21.4	1
8	40	40	40	40	50	7.99
16	53	50	64	67	70	15.96
32	72	78	82	83	90	31.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 480

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19.5	19.7	21	22	22	1
64	134	129	174	200	200	63.6
128	215	210	300	350	430	126.3
256	367	372	500	680	700	249

Language model: n-gram

# of streams	Throughput (RTFX)
32	1300

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 104

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18.3	18.3	18.6	18.7	22	1
8	38	37.3	40	50	53	7.99
16	51	50	65	66	68	15.96
32	60	68.4	70.7	71.4	80	31.9
48	88	94	98	99	113	47.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 480

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19	19.1	20	20.3	20	1
64	120	120	150	200	200	63.6
128	177	170	220	300	330	126.6
256	360	374	500	740	800	249.6

Language model: n-gram

# of streams	Throughput (RTFX)
32	1770

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 70

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	21.1	20.9	21.2	21.7	30	0.998
8	49	49	60	61	85	7.95
16	60	60	76	80	120	15.85
32	78	82	90	100	170	31.55
48	114	118	130	140	300	47

Chunk size (ms): 800
Language model: none
Maximum effective # of streams without language model (greedy generation): 300

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24	23.1	28.6	29	30	0.998
64	160	147	220	320	400	62.2
128	270	250	400	600	780	121.3
256	600	480	1000	1600	2300	229

Language model: none

# of streams	Throughput (RTFX)
32	560

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 111

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18.2	18.2	18.7	18.9	20.7	1
8	35.6	35.4	37	37	50	7.99
16	51	50	64.3	65.7	70	15.96
32	63	67.6	70	72	94	31.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 500

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19.2	19.3	20	21	23	1
64	120	120	150	160	170	63.6
128	174	180	213	220	250	127
256	295	333	395	406	450	252

Language model: n-gram

# of streams	Throughput (RTFX)
32	1400

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 100

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18.4	17.8	18.3	27	30.5	0.999
8	40	37	50	50	100	7.97
16	48	46	62.6	66	147	15.9
32	66.4	68.4	72	100	253	31.7
48	97.4	92	102	156	440	47.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 308

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	21	19	28.3	28.7	30	1
64	160	123	320	390	460	63.1
128	250	200	550	700	800	124.6
256	605	380	1480	2060	2320	242.6

Language model: n-gram

# of streams	Throughput (RTFX)
32	660

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 140

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.84	13.54	13.9	14	26.3	1
8	23	23	24	30	57	7.98
16	40.5	41	42.7	43.5	102	15.9
32	60	60	65	67	170	31.7
48	64	60	68	70	210	47.5
64	80	80	88	90	280	63.1

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 580

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14	15	22.7	24	1
64	106	96	110	207	380	63.2
128	150	135	160	300	650	125.1
256	274	250	280	610	1470	244.5
384	420	370	400	1060	2850	359
512	620	500	600	1550	4700	467

Language model: n-gram

# of streams	Throughput (RTFX)
32	2500

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 150

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.4	13.4	13.7	14	15.2	1
8	24	23	30	30	31	8
16	39	40	42	42	46	15.98
32	60	60	65	67	70	31.93
48	61.3	61.5	68	70	75	47.9
64	80	80	87	89	97	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 710

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.3	14.4	15	15.6	17	1
64	90	100	104	110	110	63.8
128	120	130	150	160	170	127.3
256	220	246	270	280	290	253.5
384	320	360	420	440	470	378.6
512	420	480	550	600	700	501
768	1500	1600	2000	2400	3000	731

Language model: n-gram

# of streams	Throughput (RTFX)
32	1000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 126

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.8	13.8	14.1	14.1	16	1
8	26	25	30	30	35	7.99
16	42	43	45	45	48	15.97
32	58	58	65	67	72	31.9
48	67	70	78	80	83	47.85
64	80	87	100	100	100	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 620

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.7	14.7	15.5	16	17	1
64	90	90	130	130	160	63.7
128	150	157	210	220	280	127
256	272	283	420	480	510	251
384	385	405	600	780	900	372.6
512	550	540	1000	1130	1500	491
768	5400	6000	9000	10000	10400	636

Language model: n-gram

# of streams	Throughput (RTFX)
32	900

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 114

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.2	15.1	15.4	15.5	18	1
8	28.7	28	29.3	30	40	7.99
16	50	53	56	56.4	60	15.97
32	64	63	70	70	75	31.9
48	85	85	90	91	97	47.8
64	104	104	113	116	140	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 560

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16.1	16	16.5	17	20	1
64	100	100	116	128	175	63.7
128	170	176	240	290	300	126.7
256	325	340	420	620	680	250.3
384	450	475	600	1000	1300	371
512	660	637	1260	1500	2000	490
768	5300	5000	8300	8650	8900	598

Language model: n-gram

# of streams	Throughput (RTFX)
32	900

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 100

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17.8	16.6	21.3	32	39	1
8	39	38.3	46	57	80	7.99
16	56	55	67	77	100	15.96
32	80	80	92	110	130	31.9
48	96	94	114	125	170	47.8
64	122	121	147	160	190	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 430

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24.5	21.6	39	39.3	40	1
64	150	150	200	200	200	63.6
128	230	230	300	320	340	126.8
256	420	420	600	640	800	250
384	640	620	1000	1200	1300	367
512	2000	2000	3000	4000	5000	480
768	7000	7000	10000	10000	12000	600

Language model: n-gram

# of streams	Throughput (RTFX)
32	600

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 110

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.4	15.3	15.8	16	17	1
8	30	30	40	40	43	7.99
16	55	56	61.4	62	70	15.96
32	64.3	63.6	70.4	73	80	31.9
48	80	84	91	93	130	47.8
64	106	105.4	113	118	170	63.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 560

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	16	16.5	16.6	18	1
64	105	104	140	150	158	63.7
128	164	177	199	205	220	127
256	300	330	370	400	400	252
384	460	500	600	700	900	373.4
512	650	670	1100	1400	1600	491
768	8000	8000	11000	11000	12000	590

Language model: n-gram

# of streams	Throughput (RTFX)
32	800

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 112

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19	18.66	19.6	20.6	30	0.999
8	25	23	30	30	58	7.98
16	34	30	40	42	93	15.93
32	46.2	44	52	55	160	31.8
48	58	53	60	64	260	47.6
64	74.5	64	72	80	440	63.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 520

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23.6	22.5	29	33.7	34	1
64	120	103	216	330	420	63.1
128	180	140	329	540	700	125.1
256	320	230	680	1190	1560	245
384	600	340	1600	2200	3200	357
512	1060	450	3350	4200	6500	456

Language model: n-gram

# of streams	Throughput (RTFX)
32	1800

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 216

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10	9.4	10.4	11.3	20.8	1
8	15	14	18	20	58	7.98
16	20	18	23.3	26	97	15.93
32	29.6	25	33	37	180	31.8
48	39	32	43	50	270	47.5
64	47	36	50	90	350	63.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 730

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14	22	22.7	24	1
64	80	57	177	300	400	63.2
128	124	78	300	530	710	125
256	220	120	570	1000	1370	245.5
384	320	160	1030	1480	2110	361
512	500	202	1600	2300	3200	468
768	1150	300	3900	5000	7400	640

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 83
Maximum effective # of streams without language model (greedy generation): 85

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	14	13.74	14.1	14.2	27	1
n-gram	8	27	26	30	32	58	7.98
n-gram	16	39	40	46	46.6	100	15.93
n-gram	32	70	72	77	78	200	31.74
n-gram	48	102	106	110	118	300	47.4
none	1	12	12	12	12	19	1
none	8	30	30	30	30	50	7.98
none	16	36	34	43	44	80	15.94
none	32	65	70	74	75	175	31.75
none	48	96	102	106.3	109	292	47.5
none	64	132	133	141	160	420	63

Chunk size (ms): 800
Maximum effective # of streams with n-gram language model: 365
Maximum effective # of streams without language model (greedy generation): 376

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	15.7	15.2	16	23.5	25	1
n-gram	64	140	137	156	210	340	63.3
n-gram	128	240	240	300	403	685	125
n-gram	256	440	440	500	780	2180	244.7
none	1	13.15	12.8	13.2	17.2	18	1
none	64	132	133	140	200	340	63.3
none	128	224	220	260	380	660	125.2
none	256	430	430	496	770	2176	245
none	384	900	687	840	2400	10370	354

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 80

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.86	12.8	13.4	13.6	16	1
8	30	30	35	35	40	7.98
16	40	40	50	50	50	15.94
32	68	75	80	80	85	31.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 370

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	14.3	15	16	16	0.999
64	130	140	150	153	155	63.3
128	240	247	270	270	280	125.4
256	430	480	530	550	560	246

Language model: n-gram

# of streams	Throughput (RTFX)
32	1100

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 83

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.2	13	13.6	13.7	19.4	1
8	25	24	30	30	50	7.99
16	36	35	43	44	78	15.98
32	70	77	82	86	156	31.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 382

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.4	14	17	20.3	20.5	1
64	130	130	140	210	330	63.7
128	222	230	255	403	650	127
256	513	434	1025	1780	2560	247

Language model: n-gram

# of streams	Throughput (RTFX)
32	620

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 80

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.8	13.86	14.26	14.34	16.2	1
8	27	27	30	30	35	7.99
16	38	36	45	50	50	15.97
32	72.4	80.5	89	90	94	31.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 383

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.5	14.64	16	16.6	18	1
64	140	140	180	200	200	63.6
128	280	270	400	440	500	126
256	500	480	900	1000	1300	247

Language model: n-gram

# of streams	Throughput (RTFX)
32	1220

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 90

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.53	13.5	13.8	13.9	16.16	1
8	25	24.3	30	30	30	7.99
16	34	33	40	42	44	15.98
32	60	70	72	73	77	31.9
48	91	102	105	106.4	120	47.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 390

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.8	13.7	14.4	15	16	1
64	140	140	160	200	240	63.5
128	230	230	300	440	500	126.2
256	460	470	700	1000	1000	248

Language model: n-gram

# of streams	Throughput (RTFX)
32	1370

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 65

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.4	15.3	15.5	16	20	0.999
8	36	40	40	41	50	7.97
16	58	60	68	73	90	15.9
32	94	100	110	120	160	31.6
48	140	145	160	170	300	47.1

Chunk size (ms): 800
Language model: none
Maximum effective # of streams without language model (greedy generation): 280

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	17.7	20.2	21	21	0.999
64	200	200	200	300	360	62.4
128	280	273	400	520	650	122.5
256	680	535	1270	1700	2500	231

Language model: none

# of streams	Throughput (RTFX)
32	630

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 85

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.4	13.5	14	14.3	16.3	1
8	26	25	30	30	30	7.99
16	35	33.6	43	44	46	15.97
32	63	70	75	76	100	31.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 414

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.6	15	16	16.3	17	1
64	124	134	145	149	154	63.6
128	220	244	260	266	270	126.7
256	400	430	500	580	700	250.3

Language model: n-gram

# of streams	Throughput (RTFX)
32	1250

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 82

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.74	13.3	15	19.6	1
8	26	24.4	27	34	56	7.98
16	38.3	40	45	48	93	15.94
32	70	70	80	100	230	31.74
48	110	107	120	200	450	47.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 290

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	13.9	20	20.4	20.7	1
64	150	135	264	316	357	63.2
128	270	240	520	620	700	125
256	670	480	1510	2160	2350	244.3

Language model: n-gram

# of streams	Throughput (RTFX)
32	670

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 133

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.5	11.4	11.6	11.8	19	1
8	18	17	19	21	38	7.99
16	30	29	32	33	67	15.95
32	50	48	50	54	123	31.8
48	68	65	70	72	200	47.6
64	80	78	86	90	265	63.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 610

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.2	11.8	12.2	17	18	1
64	83	76	91	143	280	63.4
128	150	150	170	277	540	125.6
256	292	280	320	550	1300	246.5
384	433	404	460	840	2300	363
512	600	520	630	1470	4700	476

Language model: n-gram

# of streams	Throughput (RTFX)
32	2400

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 152

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.5	10.6	11	11	12.4	1
8	17	17	19	20	22	8
16	30	30	33	35	36	15.98
32	43	42	48	50	54	31.95
48	54	54	58	60	64	47.9
64	66	66	72	73	76.7	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 690

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.5	11.7	12.4	13	14	1
64	80	80	90	93	100	63.8
128	140	147	170	170	190	127.3
256	256	280	306	310	330	253.3
384	380	410	440	450	500	378
512	500	540	640	700	1000	500
768	3300	3200	6000	7000	7000	700

Language model: n-gram

# of streams	Throughput (RTFX)
32	1300

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 107

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.05	10.07	10.3	10.36	11.5	1
8	20	20	23	24	25	8
16	30	30	35	35	37	15.98
32	53	55	60	60	70	31.9
48	70	78	84	85	90	47.8
64	92	102	110	114	130	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 500

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.5	12.48	13.1	13.5	15	1
64	100	110	120	140	200	63.7
128	203	210	300	320	386	126.4
256	360	370	600	640	660	250
384	530	530	800	1000	1400	369.5
512	900	730	1800	2000	2700	486
768	7000	7000	12000	12400	13000	530

Language model: n-gram

# of streams	Throughput (RTFX)
32	900

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 90

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.1	12	12.2	12.3	14	1
8	25	26	28	29	30	7.99
16	35	30	40	44	46	15.98
32	67	66	73	75	90	31.9
48	100	98	103	106	126	47.8
64	120	124	130	140	200	63.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 408

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	12	12.54	13	14	1
64	130	130	169	190	250	63.6
128	230	235	360	400	440	126.3
256	420	440	500	860	1000	249
384	730	673	1400	1700	2000	367.4
512	4500	5000	6600	7000	8000	423
768	13500	12500	24200	25000	26000	427.5

Language model: n-gram

# of streams	Throughput (RTFX)
32	800

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 90

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.1	13	18.3	28.6	36.1	1
8	34	33	40	50	75	7.99
16	50	50	57	70	90	15.97
32	96	96	110	123	140	31.9
48	123	124	143	153	184	47.8
64	170	160	230	270	300	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 400

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	21	18.3	35.6	36	37	1
64	146	144	190	200	210	63.7
128	260	260	330	340	400	126.7
256	460	460	700	750	930	249.5
384	800	700	1500	1700	2000	367
512	3000	3000	5000	5000	6000	470
768	10000	10700	16000	17000	17000	500

Language model: n-gram

# of streams	Throughput (RTFX)
32	700

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 89

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	11	11.3	11.4	12.2	1
8	23.5	23	26	27	30	7.99
16	38	37	45	46	47	15.97
32	67.7	67	74	76	100	31.9
48	97	96	100	100	158	47.7
64	126	124	130	135	220	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 396

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	12.1	12.7	12.8	14	1
64	120	130	142	145	150	63.6
128	220	240	258	264	270	126.7
256	440	470	550	600	700	250
384	730	710	1170	1500	1700	368
512	6000	7000	8000	9000	9000	403
768	13600	13000	23000	24000	25000	408

Language model: n-gram

# of streams	Throughput (RTFX)
32	800

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 76

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.8	12.6	13.6	14.5	21	1
8	20	20	23	25	40	7.99
16	30	29	33	36	60	15.95
32	52	49.6	55	58	150	31.8
48	73	67	74	78	300	47.6
64	122	87	99	302	995	63.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 450

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	16.2	21	24	24.5	1
64	110	94.5	160	240	310	63.4
128	200	180	300	467	600	125.5
256	390	330	670	1200	1520	246
384	830	500	2180	3000	4200	358
512	3000	670	10000	13000	15600	418

Language model: n-gram

# of streams	Throughput (RTFX)
32	1500

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 210

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	7.6	7.2	8.4	9.2	16	1
8	11.4	10.5	13	15	35	7.99
16	15.3	14.3	17.5	20	56	15.96
32	23	21	26	28	95	31.86
48	29	26	32.2	35	153	47.7
64	38	33	40	46	210	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 800

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	13	18	18	20	1
64	63	49	107	190	257	63.5
128	104	76	200	367	502	126
256	190	130	390	729	1095	248
384	290	182	713	1230	1800	366
512	420	244	1220	1850	2400	480
768	945	370	3080	4000	6040	675

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 210
Maximum effective # of streams without language model (greedy generation): 232

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	13	12.7	13	13.5	23	1
n-gram	8	23	22	25	25	47	7.98
n-gram	16	26	24.9	27	34	60	15.96
n-gram	32	42	40	50	50	110	31.83
n-gram	48	54	57	61.5	64	132	47.7
none	1	12.2	12.05	12.6	12.8	17	1
none	8	21.5	21	23	24	38	7.99
none	16	26	24	33	35	50	15.96
none	32	38	36	46.7	47.6	90	31.85
none	48	48.3	50	58.6	59.7	116	47.7
none	64	60	65	68.6	70	149	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 850

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.6	14.2	15	21	23	1
64	71	71	76.4	120	214	63.5
128	110	104	119	210	400	126.2
256	184	170	210	405	784	249

Language model: n-gram

Speaker Diarization	Language model	# of streams	Throughput (RTFX)
False	n-gram	32	1830
False	none	32	1900
True	n-gram	32	140
True	none	32	140

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 194

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.3	12.2	12.8	13	15	1
8	22.6	22.4	24	26	30	7.99
16	30	30	37	38	39	15.96
32	43	43	50	50	55	31.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 187

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.3	13.2	14	14.2	19	1
8	23	22.3	24	29	35	8
16	30	30	37	38	50	15.98
32	43	41	53	54	80	31.95

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 220

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	12	13	14	16	0.998
8	22	21	25	27	30	7.98
16	27	25	33	35	40	15.94
32	40	40	46	47	53	31.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 194

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	13	13.7	14	16	1
8	24	22.8	30	30	31	7.99
16	29	27.4	36.3	37.3	38.5	15.98
32	43	43	52	53	55	31.94

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 226

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.6	12.6	13	13	15	1
8	23	21.6	27	29	30	8
16	27	25.3	34	35	36	15.98
32	40	40	47	48	50	31.94
48	54	56.5	59	59	65	47.9

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 110

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.8	14.6	15.2	15.5	19	0.999
8	28	27	30	34	40	7.97
16	39	40	47	48	67	15.9
32	60	60.4	70	77	120	31.7
48	80	80	100	100	150	47.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 230

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.5	12.4	13	13.3	15	1
8	22	21	27	28	30	7.99
16	27	25	34	35	36	15.98
32	40	40	46	46.7	50	31.94

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 77

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25	22.4	32.4	40.6	50	0.997
8	42	37	56	64	90	7.96
16	55	50	68	86	110	15.9
32	90	90	110	120	160	31.64
48	120	120	143	160	208	47.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 250

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	36.3	39.6	66	66	66	0.997
64	190	170	300	300	300	63
128	320	300	440	480	500	124.6
256	600	570	800	900	1000	240

Language model: n-gram

# of streams	Throughput (RTFX)
32	900

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 230

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.3	12.3	12.7	12.9	15	1
8	22	21.3	25	28	29	7.99
16	27	25	34	34	35	15.97
32	38	37	46	47	50	31.93

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1040

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.7	12.6	13	15	16	1
64	62	68	72	74	82	63.8
128	94	96	115	117	200	127.4
256	150	160	200	205	220	254

Language model: n-gram

# of streams	Throughput (RTFX)
32	2390

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 150

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	13.3	16	19	19.4	0.998
8	25	24	30	32.7	40	7.96
16	32	29.4	41	50	50	15.9
32	50	50	70	83	90	31.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 630

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14	17	17	17	0.997
64	84	88	100	100	110	63
128	130	138	150	156	160	125
256	230	250	280	300	300	245

Language model: n-gram

# of streams	Throughput (RTFX)
32	1600

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 134

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	13.3	16	16	20	0.996
8	27	26	30	30	30	7.93
16	35	33	43	44	48	15.82
32	60	60	70	80	90	31.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 220

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.7	12.7	13.3	14	16	0.998
8	23	21.8	27	30	30	7.98
16	26	25	30	35	38	15.94
32	40	40	48.6	49.3	54	31.84

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 217

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	11.7	12.3	14	17.3	1
8	22	21	26	30	43	7.99
16	26	23.7	30	35	63	15.96
32	40	37	46.6	50	111	31.85
48	50	50	59	70	160	47.7

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 20

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	29.4	28.6	29.3	33	53	0.998
4	40	37.7	50	50	75	3.99
8	70	67.6	70	90	138	7.96

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 99

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	31.4	31	32	38	39	0.999
4	40	38.9	50	60	60	3.99
8	70	69	71	90	97	7.98

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	170
False	4	290
False	8	320

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 320

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.4	9.3	9.7	9.8	13	1
8	13	12.8	14	15	24.3	7.99
16	17.6	17	19.5	20.5	39	15.97
32	25	24	27	28	65	31.9
48	32	31	34	35	90	47.7
64	40	38	41	41.7	120	63.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 360

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.5	9.5	10	10	12	1
8	13	13	16	16.4	17.5	8
16	18	18	20	21	22	16
32	25	25	27	28	29.4	32
48	30	30	34	34	36	47.9
64	37	37	40	41	43	63.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 217

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.3	10.2	11	11	12	0.999
8	18.6	18	22	23	24	7.98
16	22	20	28	29	30	15.95
32	38	38.5	41	42	45	31.87
48	49	50	53	54	57	47.7
64	60	62	68	70	71	63.6

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 280

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.7	9.6	10.1	10.4	12	1
8	15.3	15	17	18	19	8
16	20	20	23.3	23.7	24.6	16
32	32	32	35	36	38	31.95
48	40	41	45	46	48	47.9
64	48	49	54	55	56.4	63.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 225

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	11	12	12	13	1
8	18	17.7	20	20	23	8
16	23	20	28.2	28.7	30	15.98
32	37	38	40.1	41	44	31.95
48	47	49	54	55	58	47.9
64	60	61	68	70	72	63.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.6	12.3	17	27.6	38	1
8	21.6	19.7	25.3	38	60	8
16	30	28	35	55	71	15.98
32	47	45	54	66	100	31.94
48	64	63	73	90	130	47.9
64	78.6	77	91	114	150	63.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 230

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	11	11.5	11.7	12.2	1
8	18	18	20	20	24	8
16	23	21	29	29.4	30.3	15.98
32	37.4	39	41.4	42	43.3	31.95
48	48.6	50	53	54	58	47.9
64	61	61.2	66.5	69	80	63.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 180

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.6	10.4	11	12	13	0.997
8	18.27	18	20	20	20	7.96
16	24	22	30	30	30	15.9
32	41	42	48	50	54	31.6
48	55	53	73	76	90	47.1
64	71	67	100	106	110	62.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 216

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.25	11.2	12	13	14	0.999
8	18	17.5	20	22	30	7.98
16	26	25	30	32	36	15.94
32	38	38.6	42	44	47	31.86
48	49.3	50.4	54	55	60	47.7
64	60	60	70	70	74	63.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 201

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.5	10	11.3	14	15.5	1
8	19	18	20	25	38	7.99
16	23	20	28	34	60	15.96
32	40	38	41.7	60	113	31.85
48	53	49	54	85	162	47.7
64	65	60	66	105	250	63.4

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 290

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.7	9.4	10.2	11.3	16.4	1
8	12	11	14	14.4	26	7.99
16	14	13	16	17	38	15.97
32	20	19	22	23.3	62	31.9
48	24	22.65	26	28	83	47.8
64	28	26	30	32	120	63.6

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 340

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	7.8	7.5	8.4	9.4	15	1
8	9	9	10.5	12	28	7.99
16	12	11	14	15	44.4	15.97
32	16	14.5	17.4	19	71	31.9
48	18	16	20	22	100	47.8
64	23	20	24	26	139	63.6

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 100
Maximum effective # of streams without language model (greedy generation): 103

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	19	19	19.4	19.65	38	0.999
n-gram	8	40	40	50	50	92	7.97
n-gram	16	50	46.6	60	66	110	15.92
n-gram	32	80	80	94	95	250	31.7
n-gram	48	112	120	126	130	320	47.4
none	1	17	17	17	17	30	1
none	8	38	37	37.6	37.9	77	7.98
none	16	50	50	60	60	100	15.93
none	32	74	80	90	90	210	31.7
none	48	105	116	119.5	122	310	47.4
none	64	138	140	150	157	430	63

Chunk size (ms): 800
Maximum effective # of streams with n-gram language model: 405
Maximum effective # of streams without language model (greedy generation): 430

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	20	20	20.7	29	29.8	1
n-gram	64	146	148	162	220	370	63.2
n-gram	128	240	250	270	410	710	125
n-gram	256	415	400	470	770	2130	244.4
none	1	17	17	18	22	23	1
none	64	133	138	150	220	370	63.2
none	128	220	210	262	400	690	125
none	256	380	360	450	750	2060	244.7
none	384	620	570	660	1570	5000	358

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 89

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	18	19	19	21	0.999
8	41	41	42.4	44	60	7.98
16	53	51	70	72	75	15.93
32	80	90	98	100	104	31.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 410

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19	19	20	20	20	0.998
64	148	160	172	180	180	63.2
128	250	270	300	310	320	125.3
256	410	450	495	510	540	246.6

Language model: n-gram

# of streams	Throughput (RTFX)
32	900

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 101

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	17	18	18	24	1
8	40	39	40	40	68	7.99
16	50	47	60	65	100	15.97
32	76	80	92	94	220	31.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 420

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	18	20	25	26	1
64	140	143	155	230	370	63.7
128	237	240	260	420	696	127
256	480	390	970	1700	2400	247

Language model: n-gram

# of streams	Throughput (RTFX)
32	570

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 90

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18.77	18.83	19.3	19.4	21.8	1
8	40.7	40.2	42.4	43	54	7.99
16	54	50	70	70	74	15.97
32	77	87	95	96	100	31.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 408

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20.2	20.4	22	22.6	23	1
64	153	160	195	200	250	63.5
128	268	267	400	420	468	126
256	480	460	800	900	1000	247

Language model: n-gram

# of streams	Throughput (RTFX)
32	1100

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 104

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	18	18	18	20	1
8	39.5	39.1	40	42	55	7.99
16	50	50	70	70	70	15.96
32	75	87	93	94	100	31.9
48	105	119	122.9	124.4	150	47.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 460

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19	19	19	20.6	22	1
64	140	147	178.5	194	234	63.6
128	235	240	310	400	430	126.3
256	380	400	500	830	967	249

Language model: n-gram

# of streams	Throughput (RTFX)
32	1530

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 60

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19	19	19	20	25	0.998
8	60	60	63	65	80	7.95
16	63	62	79	82	100	15.87
32	107	111	122	130	200	31.55
48	155	162	180	190	318	47

Chunk size (ms): 800
Language model: none
Maximum effective # of streams without language model (greedy generation): 210

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22	22	24	25	26	0.998
64	200	200	300	400	420	62
128	400	380	600	800	1000	121
256	1600	2000	2000	3000	4000	215

Language model: none

# of streams	Throughput (RTFX)
32	590

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 107

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	18	18.4	18.6	21	1
8	38.2	38	38.8	40	48.3	7.99
16	50	50	66	68	72	15.95
32	73	80	89	90	130	31.85

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 480

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	19	19.5	20.7	21	1
64	128	140	150	155	160	63.6
128	210	230	250	250	260	126.7
256	330	360	440	480	520	251

Language model: n-gram

# of streams	Throughput (RTFX)
32	1230

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 99

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	17	18	24	28	1
8	40	37	40	50	80	7.97
16	50	45	60	65	120	15.92
32	80	86	90	110	270	31.7
48	118	116.6	127.6	220	477	47.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 330

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	18	25	25	25	1
64	158	140	285	340	383	63.2
128	280	240	533	644	730	124.9
256	610	427	1420	1900	2240	244.4

Language model: n-gram

# of streams	Throughput (RTFX)
32	653

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 148

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.6	13	13	22	1
8	20.7	20	21.2	21.6	46.6	7.98
16	30	28	33	35	70	15.94
32	48	46	50	50	130	31.8
48	65	62	70	70	200	47.5
64	77	73	80	80	270	63.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 600

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.7	13	13.6	19	20	1
64	90	80	90	156	300	63.3
128	155	140	150	290	580	125.4
256	280	250	290	560	1300	246
384	416	370	423	900	2310	362
512	570	490	590	1450	4300	473

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 158

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.4	12.5	13	13	14.3	1
8	20	19.7	21	21.5	25	8
16	29	28.4	34	34.5	37	15.98
32	48.7	49	50.7	51.4	56	31.94
48	66	66	70	72	76	47.9
64	71	72	77	78	83	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 736

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	13	13.8	14.3	16	1
64	81	81	88	90	96	63.8
128	134	135	149	153	160	127.4
256	240	243	270	280	300	253.7
384	330	350	400	410	440	379
512	450	480	550	600	800	500
768	1800	2000	3000	3000	3000	725

Language model: n-gram

# of streams	Throughput (RTFX)
32	900

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 125

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.2	13.23	13.5	13.6	14.7	1
8	24.5	24.3	25.6	26	32	8
16	33	30	44	45	50	15.98
32	64	64	70	74	80	31.9
48	80	82	92	93	96	47.8
64	94	100	112	113	120	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 550

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.4	13.4	14	14.6	16	1
64	100	105	118	124	180	63.7
128	182	183	256	280	330	126.7
256	320	330	500	570	600	250.3
384	470	470	750	960	1330	371
512	780	640	1600	2000	2400	488
768	10000	10000	15300	16000	16000	565

Language model: n-gram

# of streams	Throughput (RTFX)
32	700

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 107

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.4	14	15	15	17	1
8	30.4	30.1	31.2	31.7	43	7.99
16	50	50	58	58	60	15.97
32	80	80	80	83	90	31.9
48	98	98.5	102	104	110	47.8
64	123	122	127	130	166	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 480

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	14	14.36	15	16	1
64	128	125	140	158	190	63.7
128	200	203	240	330	350	126.5
256	347	340	400	730	800	250
384	550	550	1000	1200	1660	369
512	1300	900	2700	3000	3000	486
768	16000	17000	23000	24000	25000	500

Language model: n-gram

# of streams	Throughput (RTFX)
32	700

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 80

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16.6	15	20.4	32	41	1
8	33.3	31	39	50	70	7.99
16	50	45	60	80	113	15.97
32	90	90	100	120	160	31.9
48	122	120	137	160	210	47.8
64	150	150	180	200	280	63.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 350

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23.4	21	36	38	38.5	1
64	170	160	230	240	260	63.6
128	315	310	420	450	550	126.4
256	640	600	1010	1200	1300	246.5
384	2300	2300	3500	4000	5000	357
512	7000	7000	10000	10000	12000	400
768	13800	15000	20000	22000	23000	470

Language model: n-gram

# of streams	Throughput (RTFX)
32	500

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 110

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	14	15	15	16	1
8	29.7	29.4	31	31.2	40	7.99
16	40	36	56	57	60	15.97
32	74	73	80	81	100	31.9
48	97	97	102	104	145	47.7
64	120	119	125	127	200	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 490

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	15	15.6	16	18	1
64	125	125	132	135	150	63.7
128	185	200	220	220	230	127
256	310	330	396	410	460	251.7
384	470	540	700	800	870	372.5
512	800	747	1400	2000	2000	489
768	14000	15000	20000	20000	21000	510

Language model: n-gram

# of streams	Throughput (RTFX)
32	600

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 75

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22	22	23	24	31.2	1
8	26	25	26.6	29	50	7.98
16	36	36	42.7	45	70	15.95
32	59	57	63	67	171	31.8
48	83	74	82	91	400	47.6
64	170	96	200	700	1500	63.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 420

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	27	26	33	35	50	1
64	120	104	180	270	340	63.3
128	220	187	326	508	650	125.3
256	440	356	720	1260	1800	245.6
384	920	520	2500	3500	4700	355
512	3500	710	13000	15000	17000	404

Language model: n-gram

# of streams	Throughput (RTFX)
32	1360

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 206

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9	8.6	9.8	10.5	18	1
8	12	11	13	15	43	7.98
16	16	15	19	21	67	15.95
32	22	20	25	28	114	31.83
48	29.5	27	32.5	37	167	47.7
64	38.5	34	42	50	240	63.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 780

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.5	13.7	20	20.4	22	1
64	66	50	120	210	280	63.4
128	108	76	214	393	537	125.8
256	190	120	397	750	1100	247.4
384	280	160	710	1210	1750	365
512	400	210	1180	1780	2340	479
768	820	310	2680	3560	5140	675

Language model: n-gram

# of streams	Throughput (RTFX)
32	1500

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 39
Maximum effective # of streams without language model (greedy generation): 39

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	34.4	33.5	34	34	79	0.998
n-gram	8	56	56.5	62	65	150	7.95
n-gram	16	63	60	70	70	170	15.88
n-gram	32	146	132.2	170	220	470	31.5
none	1	32.8	32.2	32.4	32.4	70	0.998
none	8	54	55	60.7	63	135	7.96
none	16	60	57	61	70	150	15.9
none	32	140	126	160	200	460	31.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 173

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	29.5	28.4	29	48	49	0.998
64	240	250	282	420	700	62.5
128	520	500	568	900	2360	122

Language model: n-gram

Speaker Diarization	Language model	# of streams	Throughput (RTFX)
False	n-gram	32	430
False	none	32	430
True	n-gram	32	90
True	none	32	90

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 39

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	33.6	33.6	34.2	34.4	36	0.998
8	58	57	64	65	70	7.96
16	63	63	66	70	80	15.93

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 32

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	36	35.6	36.3	36.6	54	0.999
8	56	56	63	63	95	7.99
16	76	75	77	78	126	15.96

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 40

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	33	32.8	33.5	34	36.5	0.996
8	57	56	62	63	66	7.95
16	59	58	62	65	73	15.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 39

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	34.7	34.7	35.3	35.6	37.5	0.999
8	58.7	58.7	64	65	67	7.98
16	62.1	62	65.1	66	80	15.96

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 40

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	33	33	33.2	33.5	36.6	0.999
8	57.4	58	62.5	63.4	66	7.98
16	60	59.3	63	64	74	15.96
32	125	130	143	145	220	31.8

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 36

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	33.1	33	33.5	33.8	36	0.998
8	65	65	70	70	80	7.96
16	72	73	83	86	100	15.92

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 41

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	33	33.1	33.5	34	35.7	0.999
8	57	57	63	65	74	7.98
16	59	58	62	70	80	15.95

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 17

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	55	52	63	70	82	0.995
8	84	79.6	100	110	155	7.93
16	154	147	170	200	300	15.78

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 70

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	56.2	58.7	87.2	90	90	0.996
64	700	650	1000	1000	1400	60

Language model: n-gram

# of streams	Throughput (RTFX)
32	200

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 41

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	26.7	26.65	26.9	27	30	0.999
8	57	57	61.5	62.4	65	7.98
16	59	58	61.4	63	80	15.96

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 196

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	32.6	32.5	33	35	35.2	0.999
64	220	251	270	277	300	63.3

Language model: n-gram

# of streams	Throughput (RTFX)
32	560

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 29

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	40.5	36.8	62.2	65	65.6	0.99
8	60	52	80	95	110	7.88
16	82	75	110	140	150	15.64

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 142

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	37.4	36.3	40	40	40	0.994
64	320	350	370	400	400	60.5

Language model: n-gram

# of streams	Throughput (RTFX)
32	400

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 34

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	28.6	28.2	31.6	32	32	0.993
8	60	55	80	80	90	7.84
16	74	70	100	110	110	15.6

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 41

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	32.8	32.7	33.3	34	36	0.997
8	57	60	60	60	70	7.95
16	60	59	64	70	70	15.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 41

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	32.5	31.8	32.4	33.8	57.8	0.998
8	56	56	61.5	63	110	7.97
16	59	57.4	61	64	120	15.94

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 3

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	61.7	59.4	61	70	129	0.996
4	3400	3300	5500	6000	7000	3.43
8	22500	22000	41000	42300	43500	3.43

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 17

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	100	100	120	120	130	0.996
4	175	172	176	200	230	3.97
8	352	350	352.6	390	418	7.9

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	46
False	4	50
False	8	50.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 64

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20.7	20	20.3	20.4	48	0.998
8	42	41.8	42.9	44	120	7.95
16	56	55	62	64	160	15.87
32	78	73	77	84	280	31.6
48	124	112	120	130	476	47.1
64	300	150	600	700	800	62.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 67

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	21.2	21.2	21.8	22	23.3	1
8	42.2	42.2	42.8	43	45.5	7.99
16	55	55	60	62	65	15.97
32	70.7	71	74	75.6	80	31.9
48	109	110	115	116	118	47.8
64	144	144	150	154	260	63.6

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 46

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24.75	24.7	25	25	27	0.998
8	55	55	59	59.4	70	7.96
16	56	56	60	60	70	15.9
32	120	119	124	160	200	31.5
48	800	800	1500	1700	2000	45
64	4300	4500	7700	8000	8000	45.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 59

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20.7	20.8	21	21.1	22.5	1
8	46	46	46.6	47	50	7.99
16	56	56	62	65	70	15.97
32	90	90	95	96	100	31.9
48	137	138	144	147	200	47.7
64	5000	4000	10000	12000	12000	59

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 46

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25.15	25.1	25.3	25.4	27.4	1
8	55	55.6	58.4	58.7	60	7.98
16	58	57	62	64	70	15.96
32	121	120	123	124	170	31.8
48	1000	1000	2000	2000	3000	46.7
64	15000	15000	25000	30000	30000	46

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 51

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23.7	22.3	27.6	38.2	46	1
8	55	53	60	75	103	7.98
16	70	70	80	100	128	15.96
32	120	120	130	150	184	31.85
48	210	190	300	318	340	47.6
64	8000	8000	14000	15000	16000	52

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 46

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25	25	25.3	25.4	26.4	0.999
8	56	55.5	59	59.7	100	7.98
16	59	57.5	63	70	104	15.95
32	123	121	123	125	220	31.76
48	840	890	1200	1300	1400	46.5
64	11000	12000	20000	20000	20000	46.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 42

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25.6	25.5	26.5	26.7	27	0.994
8	60	50	80	90	100	7.86
16	70	60	100	108	110	15.67
32	146	125	210	300	300	30.47
48	700	700	1200	1300	1400	41
64	2300	2500	3700	3800	4000	41.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 46

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	26.2	26	27	28	29.3	0.997
8	56	57	60	60	70	7.95
16	60	57.3	70	70	80	15.9
32	119	120	122	124	130	31.6
48	560	550	800	900	1000	46.2
64	4500	4700	8000	8200	8500	46.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 41

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	26.1	24.7	25.2	41.2	51.6	0.998
8	46	39	55	86	146	7.95
16	62	55.2	66.5	105	219	15.88
32	166	121	290	470	790	31.5
48	7800	7100	16100	17000	18300	41.6
64	18650	18500	29000	32000	35000	41.8

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 36

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	43.7	44.5	46	47	61	0.998
8	41.7	43.5	47.5	49	90	7.97
16	50	47	50	60	160	15.9
32	260	100	500	1200	2100	31.64

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 120

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.9	14.3	15.5	16.18	33	0.999
8	20	18.3	20.4	23.4	116	7.96
16	25.7	23.4	27	37	200	15.88
32	42	32.3	48	60	290	31.7
48	56	40	54	150	420	47.4

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 87
Maximum effective # of streams without language model (greedy generation): 88

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	13.5	13.25	13.63	13.8	26.5	1
n-gram	8	24	23	24	25	55	7.98
n-gram	16	33	30	40	41	90	15.94
n-gram	32	56	57	64	65	180	31.74
n-gram	48	90	100	100	110	300	47.4
none	1	12.3	12.14	12.3	12.4	19.5	1
none	8	22	20.8	23	24	44	7.98
none	16	27.1	26.16	27.2	30	63	15.95
none	32	51	48	60	61	160	31.76
none	48	86	90	96	100	287	47.45
none	64	118	119	123	140	420	63

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 370

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14.7	15.4	22.6	25	1
64	140	140	150	220	380	63.2
128	228	220	256	400	720	125
256	455	440	510	810	2280	243.7

Language model: n-gram

Speaker Diarization	Language model	# of streams	Throughput (RTFX)
False	n-gram	32	920
False	none	32	920
True	n-gram	32	90
True	none	32	90

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 83

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.5	13.4	14	14.5	17	0.999
8	26	25	30	30	33	7.98
16	40	40	50	50	50	15.95
32	68	73	80	81	83	31.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 63

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	14	14.6	14.7	21.7	1
8	27	26	28	29	43	8
16	37	40	40	46	65	15.98
32	73	80	86	87	139	31.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 87

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.9	12.7	13.4	14	17	0.998
8	22	22	24	25	30	7.98
16	33	31	40	41	45	15.93
32	54	60	64	66	75	31.77

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 82

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.7	13.7	14.2	14.4	16	1
8	28	28	30.1	31	33	7.99
16	38	38.3	45	47	50	15.97
32	70	80	85	87	90	31.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 86

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.9	13.2	13.3	17	1
8	23	23	24	27	30	7.99
16	31.5	30	40	40	43	15.97
32	60	64	70	70	74	31.9
48	95	103	108	110	140	47.8

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 60

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.6	15.4	15.8	16	20	0.999
8	38	40	44	48	59	7.96
16	53	54	64	66	90	15.9
32	89	96	104	110	170	31.6
48	131	140	150	160	300	47

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 90

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.77	12.8	13.3	13.5	15.6	1
8	23	22.5	24	26	30	7.99
16	33.6	31.8	41	42	45	15.97
32	58	64	67	68	90	31.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 33

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	28	25	36	44	56	0.997
8	54	52	67	80	106	7.95
16	80	76	100	110	140	15.86
32	177	172	220	250	300	31.5
48	6300	6400	10000	10200	11000	34.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 140

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	39.5	42.4	72	74	74	0.997
64	330	344	410	440	500	62.3
128	700	680	1000	1200	1300	120
256	9000	9000	12000	13000	13000	150

Language model: n-gram

# of streams	Throughput (RTFX)
32	500

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 89

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.9	12.8	13.3	13.4	16	1
8	23	22	30	30	30	7.99
16	31	29	39	39.6	41.6	15.97
32	56	63	66.3	67	70	31.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 420

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.7	13.6	15	16	16	1
64	120	130	140	146	154	63.7
128	210	220	250	260	300	126.6
256	364	413	467	477	500	251.4

Language model: n-gram

# of streams	Throughput (RTFX)
32	1300

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 56

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.7	14.8	18.5	22	22.2	0.997
8	29	27	35	39	40	7.95
16	43	40	50	70	80	15.84
32	89	86	150	180	200	31.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 270

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	16	20	20	20	0.997
64	180	190	205	210	210	62
128	340	350	400	400	400	121.3
256	580	630	720	740	800	231

Language model: n-gram

# of streams	Throughput (RTFX)
32	790

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 66

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.3	14	17	17	18	0.996
8	30	30	30	40	40	7.93
16	50.5	51.5	60	70	74	15.75
32	87	90	130	140	150	31

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 90

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.9	13.6	15	16	0.998
8	23.6	23	26	28	31.4	7.97
16	34	34	40.7	41.4	45	15.93
32	57	62.2	65	67	74	31.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 82

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.9	12.63	13.16	15	19.6	1
8	23	22	26	30	56	7.98
16	34	35	39.4	43	96	15.94
32	60.5	62.6	66.4	87	233	31.74
48	102	98	116	170	447	47.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 7

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	35	34	34.6	40	70	0.998
4	76	73	74	90	170	3.98
8	1300	1300	2000	2500	3000	7.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 37

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	36	35.8	36.7	44	46	0.999
4	75	74	80	90	100	3.99
8	156.4	154.5	158	180	200	7.95

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	94
False	4	115
False	8	117

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 121

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.25	11.07	11.3	11.46	19.3	1
8	16	16	17	20	36	7.99
16	28	28	30	30	65	15.95
32	46	44	50	50	124	31.8
48	60	60	67	70	200	47.5
64	84	79	88	90	290	63.1

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 128

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.8	10.8	11.3	11.5	12.9	1
8	16.3	16	19	20	20.85	8
16	26	26	28	29	34	16
32	43	42	50	50	55	31.95
48	60	60	66	70	70	47.9
64	70	70	80	84	90	63.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 66

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.9	11.87	12.3	12.6	14	0.999
8	20	20	22	23	27	7.98
16	38	40	45	46	46	15.94
32	67	68	72	73	76	31.76
48	112	110	126	140	170	47.3
64	200	170	300	300	370	62.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 96

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.5	11.5	11.8	11.9	13	1
8	18	17.5	22	23	24	8
16	26	25	32	32	34	15.98
32	50	50	53	53.6	56	31.94
48	78.6	79	84	86	90	47.9
64	100	100	110	114	130	63.7

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 65

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.1	12.04	12.3	12.5	14	1
8	20	20	30	30	30	7.99
16	30	27.4	38.4	39	40	15.98
32	70	70	70	70	80	31.9
48	118	120	126	130	150	47.75
64	180	172	220	300	340	63.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 80

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.6	13.2	19	30	37	1
8	33	32	38	53	76	7.99
16	50	50	56.3	70	93	15.98
32	84	80	100	110	130	31.9
48	120	120	137	150	180	47.8
64	153	150	190	220	270	63.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 70

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	12	12.3	12.5	14	1
8	25	20	30	30	32	7.99
16	40	42	45	45	47	15.97
32	72	72	77	79.6	100	31.86
48	110	110	125	130	200	47.7
64	134	127	170	250	400	63.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 64

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.4	12.3	13.2	13.5	15	0.997
8	21	20.5	23	24	28	7.96
16	30	28	40	39	40	15.87
32	71	67	90	102	110	31.24
48	130	114	180	200	240	46
64	230	180	360	400	500	60.1

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 69

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.1	12	12.45	13	15	0.999
8	21	20	23	25	28	7.98
16	40	40	44	44	49	15.93
32	65	65	70	73	80	31.8
48	108	108	117	120	130	47.4
64	142	147	162	170	200	63

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 63

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.1	11.8	12.2	17.6	18.2	1
8	23	21	25	32	51	7.98
16	40	37.5	40	60	99	15.93
32	72	65	76.5	100	245	31.74
48	125	105	140	290	480	47.4
64	350	160	900	1280	1900	62.8

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 67

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16.6	16.3	17.4	18.4	25	1
8	21.4	20.6	23	24.5	44	7.98
16	34	30	38	40	67	15.95
32	59	55.7	66	70	180	31.8
48	96.4	80	92	110	600	47.6
64	430	250	940	1700	2800	63

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 230

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	7	6.7	7.84	8.7	15.1	1
8	11	10	12	12.5	37	7.99
16	14	13	16	18	60	15.96
32	21.7	19.7	23.8	26.5	110	31.84
48	27	24	30	33	166	47.7
64	34	29	36	42	244	63.4

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 196
Maximum effective # of streams without language model (greedy generation): 220

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	11	10.9	11.3	11.5	20	1
n-gram	8	21	20.5	22	23	43	7.99
n-gram	16	28	27	33	34	64	15.96
n-gram	32	40	43	46	48	100	31.85
n-gram	48	53	55	60	60	128	47.7
none	1	10.4	10.3	10.4	10.5	15	1
none	8	19	18.8	19.4	19.6	32	7.99
none	16	22	22	23	23	41	15.97
none	32	34.7	33	41	42	81	31.9
none	48	47	50	54	54.7	103	47.7
none	64	58	61	63	65	135	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 790

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.7	12.4	12.9	19.2	20	1
64	75	72	82	130	200	63.5
128	113	105	140	200	360	126.4
256	180	180	220	370	685	250

Language model: n-gram

Speaker Diarization	Language model	# of streams	Throughput (RTFX)
False	n-gram	32	2100
False	none	32	2100
True	n-gram	32	120
True	none	32	110

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.2	11.1	11.8	11.96	14	1
8	22	22	24	26	30	7.99
16	29	29	34	35	37	15.96
32	40	42	50	50	56	31.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 156

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.3	11.2	11.7	12	17	1
8	21	20.5	23	24	34	8
16	28	30	33.7	34.4	45	15.98
32	43	47	49	50	70.4	31.95

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 190

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.7	10.5	11	12	14	0.999
8	19.14	19	20	21	24	7.98
16	25	30	30	30	33	15.94
32	35	36	42	43	47	31.85

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 170

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.3	11.3	11.9	12	13.9	1
8	22	22	24	25	30	8
16	28.4	28.5	35	36	38	15.98
32	44	47	54	55	60	31.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 220

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	10.9	11.3	11.4	13.7	1
8	18.2	17.8	20	20	24	8
16	23.5	23	28	29	31.7	15.98
32	35	39	41.15	41.9	48	31.95
48	48	51	54	55	64	47.9

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 100

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.2	13	13.4	14	17	0.999
8	26	26.3	28	30	37	7.98
16	37	37	45	50	60	15.92
32	56	60	70	72	100	31.75
48	80	85	96	100	150	47.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 200

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.4	10.3	10.8	11	13	1
8	19.3	19.2	20	20	23	8
16	24	23	30	30.6	33	15.98
32	36	40	43	44	47	31.94

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 68

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16.95	14.3	24.4	33	42	0.997
8	33	29.2	46	64.4	82	7.96
16	50	50	60	80	100	15.9
32	80	80	100	110	140	31.7
48	120	127	160	180	200	47.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 280

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	29	33	59	59	59	0.997
64	180	175	260	300	400	63
128	320	300	450	460	500	124.6
256	660	630	900	1000	1300	240

Language model: n-gram

# of streams	Throughput (RTFX)
32	932

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 225

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.46	10.4	10.7	10.9	13.6	1
8	18	17.4	20	20	23.6	7.99
16	23	22	28	29	32	15.98
32	35	38	40.8	42	50	31.94

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 930

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.6	11.4	13	14	15	1
64	64	70	76	77	80	63.8
128	90	93	110	120	130	127.4
256	160	175	213	220	240	253.7

Language model: n-gram

# of streams	Throughput (RTFX)
32	2600

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 130

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.6	12	16	16.4	16.7	0.998
8	22.5	21.8	27	28.6	30	7.96
16	30.6	30	37	39	40	15.9
32	52.4	52	83	88.5	93	31.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 570

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.5	13	16	16	16	0.998
64	87	90	104	107	110	63
128	160	160	190	200	200	124
256	260	280	320	400	400	243

Language model: n-gram

# of streams	Throughput (RTFX)
32	1700

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 130

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	11.8	14.7	15	16	0.996
8	22	22	25	27	30	7.94
16	31	30	37	40	43	15.87
32	50	50	75	85	90	31.44

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 200

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.6	10.6	11.1	12	13.4	0.999
8	19.6	19.4	21	23	24	7.98
16	26.5	25	31.7	32.6	40	15.94
32	38	40	46	48	60	31.84

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 195

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.6	10.3	11	13	17	1
8	20	19.5	21	25	41	7.99
16	27	27	31.5	34	62	15.96
32	40	40	43	50	104	31.86
48	50	50	55	70	145	47.7

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 18

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18.7	18	19.3	23	33	0.999
4	31.4	30	31.5	40	64	3.99
8	59	57	59	71	130	7.96

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 91

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	21	20.5	22	27	30	1
4	32.6	32	33.4	43	50	3.99
8	60	58	61	77	88	7.98

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	194
False	4	270
False	8	280

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 256

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	8.7	8.6	8.8	9	13.6	1
8	15	15	16	16.3	28	7.99
16	20	20	23	24	46	15.96
32	32	32	33.6	35	80	31.9
48	40	40	44	46	100	47.7
64	53	51	57	60	140	63.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 240

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	8.34	8.4	8.7	8.9	10.1	1
8	17	17	18.3	19	23	8
16	23	23.5	26	27	30	16
32	35	36	38	38	40	31.96
48	46	46	52	53.1	56	47.9
64	55	56	62	64	70	63.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 187

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.4	9.4	9.7	10	10	0.999
8	16.5	16.4	18	18	20	7.99
16	20	20	23	26	28	15.96
32	37	37.6	40	40	42	31.9
48	49	49	53	55	60	47.7
64	59.7	60.4	65	66	70	63.6

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 217

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.1	9.1	9.5	9.6	10.7	1
8	15.8	15.8	17	18	20	8
16	24	25	26	27	30	16
32	34	34	37	39	42	31.95
48	46	46	50	52	60	47.9
64	54	55	60	60	70	63.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 190

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.54	9.5	9.66	9.7	12	1
8	17	16.4	18	20	20	8
16	22	22	25	27	28	16
32	37	38	40	41	47	31.95
48	48	49	52	54	60	47.9
64	59.5	60	64.6	66	70	63.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.4	11	16.6	27.5	34.6	1
8	20	18	24.7	42	67	8
16	28	26	33	53	76	15.98
32	47	44	60	70	100	31.94
48	61	60	70	95	114	47.9
64	75	73	86	100	142	63.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 196

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.2	9.2	9.6	9.8	10.6	1
8	17	16.4	18	19	20	8
16	23	23	26.7	27.4	30	15.98
32	36.6	37	39.5	40.5	43	31.95
48	48	49	52	53	56	47.9
64	60	60.5	65	67	82	63.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 156

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.7	9.6	10	10	10	0.997
8	17.4	17	19	20	23	7.96
16	22.3	22.3	23.7	27	30	15.9
32	43	45	50	60	60	31.6
48	60	60	80	90	100	47
64	80	76	110	120	120	62.3

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 187

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.6	9.5	10	10	12	0.999
8	18	17	20	20.7	24	7.98
16	28	28	30.4	31	35	15.95
32	38	39	42	45	50	31.85
48	49.4	50.5	56	58	65	47.7
64	60	62	69.4	71.6	81	63.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 180

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.3	9.1	9.5	12.8	13.2	1
8	17.4	16.5	20	22.7	34	7.99
16	27.3	26.6	28.6	36	60	15.96
32	40	38	42	57	100	31.87
48	52	49	54	81	141.5	47.7
64	64	60	66	100	220	63.5

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 130

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.1	10.8	12	13	18	1
8	16.7	16	19	20	31	7.99
16	23.7	23	27	28.3	46	15.97
32	37	36	41	42.5	75	31.9
48	46	45	51	53.6	99	47.8
64	57	55	61	65	170	63.6

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 280

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	6.9	6.6	7.6	8.4	14	1
8	9.2	8.7	10	11	27	7.99
16	13	12.4	15	17	41	15.97
32	17.5	17	20	23	68	31.9
48	23	22	28	30	94	47.8
64	27	25	30	36	120	63.7

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	27.6	29.14	29.95	53.7	28.16	0.995

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	26.64	28.2	31.86	37.6	26.93	0.994

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	25.63	26.66	27.36	33.75	25.53	0.997

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	25.43	27	34.16	35.2	26.43	0.99

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	27.4	28.7	29.4	40.6	27.4	0.997

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	25.6	26.3	26.5	33.9	25.86	0.997

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	24.4	24.84	24.93	47.6	24.9	0.997

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	26.2	27	29.7	40.2	26.27	0.995

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	41.6	42.24	45.1	61.2	42.4	0.99

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	23.93	24.5	24.6	33.05	24.15	0.997

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	29.1	40.8	44.6	44.6	31.3	0.98

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	26.5	41.05	41.05	41.05	29	0.974

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	26.4	28.1	33.7	33.7	26.94	0.989

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	25.2	26.3	29.36	39.4	25.55	0.996

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	18.37	19.05	19.36	29.44	18.73	0.997

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	18.07	18.86	18.9	27.8	18.1	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	22.7	23.38	25.56	32.5	23.2	0.993

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	18.47	19.66	21.44	29.3	18.66	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	22.75	23.17	23.66	32.16	22.98	0.997

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	19.14	34	52.7	67.6	22.96	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	23.97	25	27.27	34.8	24.03	0.996

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	23.57	33.93	33.93	33.93	24.86	0.981

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	23.73	24.7	30.45	32.3	24.06	0.991

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	22.4	23.24	24.5	32.9	22.6	0.997

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	40.9	44.4	48.1	184	44	0.992

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	41.6	100	184.6	189.6	52.8	0.992

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	37.8	40.55	44.4	50.7	37.7	0.989

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	64.4	592	713	754	192.5	0.98

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	38.5	39.8	40.5	58.4	38.9	0.994

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	36.1	37.9	39.3	73.6	37.06	0.995

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	201.8	888	1032	1130	290	0.994

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	73.3	76	79.7	136.6	76	0.952

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	34.64	35.4	36.25	47.6	34.96	0.996

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	45.5	72.4	77.7	77.7	50.4	0.967

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	2573	2806	2806	2806	2360	0.3496

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	41.4	50.4	52.1	54.7	42.2	0.984

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	37.8	169.7	325	570	70.4	0.986

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	29.77	31.9	32.5	78.1	30.94	0.995

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	31.44	192.5	227	306	69.3	0.996

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	40.24	188.7	189.4	254.6	90.7	0.965

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	35.6	37.6	41.3	52.8	36.3	0.995

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	1790	3250	3410	3600	1598	0.769

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	718	1560	1630	1843	742	0.978

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	1007	1260	1260	1260	847	0.711

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	37.7	48.9	55.7	176.7	44.9	0.986

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	34.6	54.8	183	432	51.8	0.994

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	14.2	14.87	14.96	99	15.45	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	14.4	15.97	17	99.8	17.36	0.997

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.48	14.38	14.8	102.8	14.8	0.999

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.65	15.34	17.65	99.3	17.64	0.995

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	14.16	14.8	15	93.2	15.52	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	14.1	14.55	14.83	93.7	15.23	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	15.3	16.17	16.66	96.4	16.78	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.97	15.1	17.28	98.8	15.5	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	24.3	25.75	62	219	32.7	0.994

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.26	14.38	14.97	192.7	16.73	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	16.1	20.7	98	98	21.95	0.99

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	14.43	97	97	97	23.05	0.987

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.23	14.88	17.96	155.8	19.73	0.995

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.46	14.33	16.06	102	14.92	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	11.15	12.1	12.56	187	14.06	0.999

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	11.15	11.96	12.31	103	12.6	0.999

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.2	14.22	14.44	94	16.82	0.995

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	11.35	12.02	12.66	91.2	12.73	0.999

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.26	14.07	14.28	107.1	15.36	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.16	20.66	36.24	96.5	16.22	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	12.77	13.44	14.02	100.8	14.3	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.03	93.8	93.8	93.8	20.98	0.99

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.63	14.98	17.28	93.8	17.24	0.995

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	12.98	13.47	13.72	98	14.2	0.999

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 33

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	50	49.6	50.1	50.4	100	0.997
8	70	70	80	80	130	7.96
16	100	100	111	114	230	15.87
32	300	280	450	500	660	31.4
48	12100	13000	23200	24000	24700	34.2
64	22000	24500	38400	39100	39700	34

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 150

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	52.6	52	53	64.3	67	0.998
64	300	320	340	550	700	62.7
128	580	570	680	900	2000	123.2
256	17000	19000	30000	30000	38000	154
384	31000	32600	50000	53000	62600	155.5
512	45500	47000	71000	73000	86800	159

Language model: n-gram

# of streams	Throughput (RTFX)
32	423

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 52

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22.3	21.76	22	22	42.6	0.999
8	50	48	50	60	100	7.97
16	57	56	60	63	125	15.9
32	100	95	100	100	280	31.7
48	170	145	200	330	500	47.3
64	10000	10000	19000	20000	20400	52.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 236

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24	23.4	23.8	32	34	0.999
64	185	180	190	300	370	63.2
128	360	370	386	510	740	124.8
256	1300	800	1800	2200	12000	240
384	17000	18000	27400	27800	34700	246
512	28000	32000	40500	41000	50000	250
768	45000	50000	68000	70000	80000	256

Language model: n-gram

# of streams	Throughput (RTFX)
32	1100

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 36

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	49	48.6	49	49	100	0.997
8	70	80	85	85	120	7.97
16	90	100	100	102	200	15.9
32	300	286	400	485	780	31.5
48	10000	11000	21000	22000	23000	36.6
64	20300	23600	35400	36400	37000	37

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 173

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	52	51.2	52	63.3	65	0.998
64	300	300	310	400	500	63
128	520	550	600	790	1500	124
256	14000	15000	25000	26000	35000	172
384	27000	27000	44000	48000	58000	174
512	39000	38000	6

NVIDIA Riva

Performance

Contents

Performance#

Evaluation Process#

Results#

Speaker Diarization	Language model	# of streams	Throughput (RTFX)
False	n-gram	32	1020
False	none	32	1000
True	n-gram	32	100
True	none	32	100