Performance#

Evaluation Process#

This section shows the latency and throughput numbers for streaming and offline configurations of the Riva ASR service on different GPUs. These numbers were captured after the pre-configured ASR pipelines from our Quick Start scripts were deployed. The Jasper, QuartzNet, Conformer, and Citrinet-1024 acoustic models were tested.

In streaming mode, the client and the server used audio chunks of the same duration (100ms, 160ms, and 800ms depending on the server configuration). Refer to the Results section for the chunk size value to use.

The Riva streaming client riva_streaming_asr_client, provided in the Riva image, was used with the --simulate_realtime flag to simulate transcription from a microphone, where each stream was doing three iterations over a sample audio file (1272-135031-0000.wav) from the LibriSpeech dev-clean dataset. The LibriSpeech datasets can be obtained from https://www.openslr.org/12.

The source code for the riva_streaming_asr_client can be obtained from https://github.com/nvidia-riva/cpp-clients.

The command used to measure performance was:

riva_streaming_asr_client \
   --chunk_duration_ms=<chunk_duration> \
   --simulate_realtime=true \
   --automatic_punctuation=true \
   --num_parallel_requests=<num_streams> \
   --word_time_offsets=true \
   --print_transcripts=false \
   --interim_results=false \
   --num_iterations=<3*num_streams> \
   --audio_file=1272-135031-0000.wav \
   --output_filename=/tmp/output.json

The riva_streaming_asr_client returns the following latency measurements:

intermediate latency: latency of responses returned with is_final == false
final latency: latency of responses returned with is_final == true
latency: the overall latency of all returned responses. This is what is tabulated in the following tables.

Refer to the following diagram for a schematic representation of the different latencies measured by the Riva streaming ASR client.

Schematic Diagram of Latencies Measured by Riva Streaming ASR Client

In offline mode, the command used to measure maximum throughput was:

riva_asr_client \
   --automatic_punctuation=true \
   --num_parallel_requests=32 \
   --word_time_offsets=true \
   --print_transcripts=false \
   --num_iterations=96 \
   --audio_file=1272-135031-0000x5.wav \
   --output_filename=/tmp/output.json

where 1272-135031-0000x5.wav is simply the 1272-135031-0000.wav audio file concatenated five times. The source code for the riva_asr_client can be obtained from: https://github.com/nvidia-riva/cpp-clients

Results#

Latencies and throughput measurements for streaming and offline configurations are reported in the following tables. Throughput is measured in RTFX (duration of audio transcribed / computation time).

Note

Audio files were iterated 1 time for Xavier AGX, Xavier NX, and Orin AGX and 3 times for all other experiments.

Note

If the language model is none, the inference is performed with a greedy decoder. If the language model is n-gram, then a beam decoder was used.

Note

The values in the tables are average values over 3 trials. The values in the table are rounded to the last significant digit according to standard deviation calculated on 3 trials. If a standard deviation is less than 0.001 of the average, then the corresponding value is rounded as if standard deviation equals 0.001 of the value.

For specifications of the hardware on which these measurements were collected, refer to the Hardware Specifications section. Please notice, that

results on AWS and GCP are computed using Riva 2.4.0
results On-Prem on GPUs A40, T4, L40, H100 on Riva 2.9.0.
other On-Prem results on Riva 2.8.0

Cloud instance descriptions for AWS and GCP.

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 180
Maximum effective # of streams without language model (greedy generation): 192

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	13.9	13.57	13.9	14.2	28	1
n-gram	8	26.2	25.5	26	27	60	7.98
n-gram	16	37	35	45	45.4	90	15.93
n-gram	32	52	56	63	64	140	31.8
n-gram	48	62	65	70	72	150	47.6
n-gram	64	76	76	81	83	230	63.3
none	1	13.1	12.8	12.96	13.2	22	1
none	8	25	24.5	25	25	60	7.98
none	16	34	30	44	44	80	15.94
none	32	50	50	61	63	120	31.8
none	48	56	62	66	70	140	47.6
none	64	70	73	76	77	190	63.3

Chunk size (ms): 800
Maximum effective # of streams with n-gram language model: 690
Maximum effective # of streams without language model (greedy generation): 750

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	16	15.4	16.3	24	30	1
n-gram	64	97	96	107	170	300	63.3
n-gram	128	140	130	168	270	530	125.7
n-gram	256	230	220	270	530	1170	247
n-gram	384	330	330	380	740	2000	365
n-gram	512	470	440	500	1240	3060	478
none	1	13	12.67	13	17.1	18	1
none	64	86	88	93	160	290	63.4
none	128	120	100	140	250	500	125.9
none	256	200	200	240	470	1090	247.7
none	384	300	304	340	690	1700	365.6
none	512	420	410	460	1040	2870	480

Speaker Diarization	Language model	# of streams	Throughput (RTFX)
False	n-gram	32	1500
False	none	32	1600
True	n-gram	32	120
True	none	32	120

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 150

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.8	13.7	14.3	14.6	20	1
8	26.6	26.3	27.7	30	40	7.98
16	37	36	46	47	50	15.95
32	57	63	70	70	73	31.84
48	67	73	80	80	90	47.7
64	80	87	94	95	100	63.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 740

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14.5	15.6	20	20	0.999
64	95	103	111	114	117	63.4
128	150	150	200	200	200	126.2
256	220	235	270	300	300	250.5
384	300	340	400	410	440	372
512	400	470	540	560	650	492
768	660	730	1000	1000	1200	720

Language model: n-gram

# of streams	Throughput (RTFX)
32	1500

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 182

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.5	13.3	13.8	14	20	1
8	25.7	25	25.7	27	52	8
16	40	30	44	45	80	15.97
32	50	50	62	63	127	31.95
48	61.6	65	70	74	183	47.9
64	74.6	75	81	87.7	270	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 704

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14	20	21	20	1
64	93	94	110	175	300	63.8
128	140	130	160	300	530	127.4
256	228	210	270	585	1190	253.5
384	390	320	780	1300	2100	371
512	620	430	1720	2440	3400	483
768	1500	660	4400	7300	10000	682

Language model: n-gram

# of streams	Throughput (RTFX)
32	1000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.9	13.9	14.3	14.4	17	1
8	27.5	27.3	28.3	30	40	7.99
16	40	37	40	50	50	15.98
32	57	64	70	71	74	31.9
48	70	76	81.4	83	90	47.9
64	86.6	93	99.4	101	110	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 760

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14.8	16	17	20	1
64	90	100	110	115	140	63.7
128	140	135	180	200	270	127
256	230	230	330	400	430	252
384	340	340	646	700	800	374
512	460	470	800	970	1200	494
768	920	740	1900	2300	3000	727

Language model: n-gram

# of streams	Throughput (RTFX)
32	1900

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 188

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.5	13.37	13.7	13.8	20	1
8	27	25	30	30	30	7.99
16	34	30	44	44.5	50	15.97
32	50.3	57	62	63	69	31.9
48	59	65	68	70	80	47.9
64	70	76	80	81	90	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 800

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14.3	15	16	20	1
64	85	93	103	108	120	63.8
128	130	120	160	200	230	127.2
256	197	200	260	400	500	252.3
384	285	310	400	650	670	375.5
512	390	420	600	860	1000	496

Language model: n-gram

# of streams	Throughput (RTFX)
32	2800

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 90

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.1	14.9	15.4	16	20	0.999
8	34	34	40	40	56	7.97
16	48	48	60	60	90	15.9
32	73	77	90	90	150	31.6
48	92	100	106	110	200	47.25
64	110	115	125	140	270	62.8

Chunk size (ms): 800
Language model: none
Maximum effective # of streams without language model (greedy generation): 380

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18.3	17.6	20	20	20	0.999
64	130	124	160	240	300	62.7
128	220	200	300	500	700	123
256	400	370	700	1000	1300	237
384	680	556	1350	2000	2600	337
512	2000	2000	4000	6000	8000	400

Language model: none

# of streams	Throughput (RTFX)
32	870

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 190

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.6	13.48	14	14.5	20	1
8	25.5	25.4	26.2	26.7	34	7.99
16	35	32	44.4	45	48	15.97
32	50	56	63	63	70	31.9
48	58	64	69	70	97	47.8
64	70	74	79	82	118	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 860

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14.5	15	16	20	1
64	85	92	100	100	110	63.7
128	110	110	140	150	170	127.2
256	172	184	237	243	255	253.4
384	246	295	341	350	380	378.6
512	340	410	460	494	540	502
768	540	608	800	900	1150	740

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 57

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	31.7	30.6	32.9	33.1	50	0.996
8	54	52.4	56.5	59.3	100	7.94
16	68	67	74	86	110	15.87
32	120	124	130	140	170	31.6
48	200	200	250	300	330	47
64	2000	2000	4000	4000	5000	58

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 250

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	36	35.8	50	50	50	0.996
64	200	210	300	300	400	62.7
128	400	400	460	470	600	123.5
256	1000	900	1600	2000	2000	230
384	5000	5000	7000	7000	8000	285
512	8000	8000	10000	12000	12000	300

Language model: n-gram

# of streams	Throughput (RTFX)
32	700

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 175

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.5	13.13	13.7	17	20	1
8	26	24.7	26	32	57	7.98
16	34	30	44	46	94.5	15.93
32	50	50	62	80	160	31.8
48	63	63	70	82	243	47.6
64	78	75	83	128	300	63.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 510

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14	20	20	20	1
64	108	86	230	284	323	63.3
128	170	120	400	500	570	125.6
256	310	223	720	1080	1250	247
384	530	340	1350	1640	2150	364.4
512	1200	600	3000	3600	4300	468

Language model: n-gram

# of streams	Throughput (RTFX)
32	1000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 12

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	38	36.5	37.1	43	72	0.998
4	55.5	53	58	64	112	3.985
8	106.6	102	102.7	123	250	7.94

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 62

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	40	38.2	39	46.4	70	0.999
4	69.2	68.3	69.5	81.4	90	3.99
8	106	103	110	120	140	7.97

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	130
False	4	183
False	8	190

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 250

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.9	9.7	10	10.15	16.7	1
8	14.5	14	15	15.5	34	7.99
16	23	23	25	25.8	60	15.95
32	35	34	36	37	100	31.8
48	46	44	50	50	140	47.6
64	55	53	58	60	200	63.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 980

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	10.2	11	16	20	1
64	64	56	60	135	260	63.4
128	108	94	110	240	470	126
256	172	148	174	400	960	248.3
384	240	207	250	585	1430	367
512	320	270	320	800	2100	482
768	484	400	480	1400	3900	703

Language model: n-gram

# of streams	Throughput (RTFX)
32	4000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 260

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.53	9.53	9.8	9.9	13	1
8	13.7	13.6	14.4	15	19	8
16	20	20	24	25	27	16
32	34.5	34.6	36	37	41	31.96
48	43	43	47	48	51	47.9
64	54	54	60	60	68	63.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1240

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.3	10.3	11	11.5	20	1
64	55	55	62	65	70	63.9
128	96	96	108	112	120	127.5
256	150	160	190	200	220	254.4
384	190	203	240	240	260	381
512	240	265	306	316	340	506
768	350	397	460	480	520	755

Language model: n-gram

# of streams	Throughput (RTFX)
32	1000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 230

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10	9.9	10.3	11	14	1
8	16	16	17	20	23	8
16	24	25	30	29.5	32	15.98
32	42.5	42.4	44.5	46	50	31.94
48	53	53	60	63	66	47.9
64	60	60	66	70	74	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1000

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	10.7	11.4	12	20	1
64	69	70	80	80	86	63.8
128	100	104	140	150	200	127.2
256	176	184	260	290	330	253
384	260	260	400	460	600	377
512	310	334	500	610	645	499
768	500	500	800	1060	1400	737

Language model: n-gram

# of streams	Throughput (RTFX)
32	1000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 190

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.2	11.07	11.3	11.4	15	1
8	20	20	20.8	21	30	8
16	30	30	37	38	45	15.98
32	52	53	55	56	64	31.9
48	64	64	70	71.7	75	47.9
64	70	72	78	79	85	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 830

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.4	12.2	13	13.5	20	1
64	88	90	97	100	110	63.8
128	140	130	170	200	240	127
256	210	220	270	423	460	252
384	290	314	380	620	670	375.6
512	382	410	530	900	960	496
768	670	626	1350	1700	2200	733

Language model: n-gram

# of streams	Throughput (RTFX)
32	1000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 150

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.2	11.7	17	29	35.6	1
8	26	23	30	56	80	8
16	38	37	45	70	94	15.97
32	67	65	74	90	114	31.9
48	87	85	100.8	120	160	47.8
64	100	97	110	130	180	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 600

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20.5	17.7	34	35	40	1
64	116	110	160	170	200	63.8
128	190	180	260	270	300	127
256	300	310	400	420	600	252.5
384	460	470	700	750	900	374
512	600	610	900	1100	1200	492
768	1300	1300	2000	2300	3000	725

Language model: n-gram

# of streams	Throughput (RTFX)
32	600

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 180

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.1	11.05	11.46	11.85	20	1
8	19.2	19	20.1	20.4	25	8
16	30	30	38	38	43	15.97
32	50	50	55	57	70	31.9
48	63	61.6	70	73	90	47.85
64	70	70	78	80	100	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 850

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	11.9	12.5	12.7	20	1
64	86	88	97	100	105	63.8
128	120	120	150	150	160	127.3
256	184	210	236	240	260	253.4
384	255	300	340	344	356	379
512	343	400	460	510	600	502
768	570	620	900	1000	1400	740

Language model: n-gram

# of streams	Throughput (RTFX)
32	900

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 175

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.14	12.85	13.8	15	21	1
8	15.2	14.6	16.4	18.1	37	7.99
16	20	20	25	27	59	15.95
32	34	33.5	39	42	98	31.84
48	43	41	48	52	165	47.7
64	50	46	56	60	235	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 820

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17.5	17	22	25	27	1
64	72	57	130	210	280	63.4
128	123	96	230	380	500	126
256	200	140	400	700	1040	248
384	279	186	640	1140	1500	367.4
512	390	240	1110	1500	2180	483

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 290

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	7.1	6.8	8	9	16	1
8	10	9.2	11	12	37	7.99
16	14	13	16	18.5	60	15.96
32	18	16	20	24	100	31.85
48	23	19	26	30	154	47.7
64	28	23	30	37	196	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1000

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	11.75	16.5	17	20	1
64	55	40	110	197	263	63.4
128	85	56	190	350	470	126
256	146	86	350	656	940	248.6
384	207	110	550	993	1360	368
512	280	146	840	1320	1900	484
768	500	200	1600	2330	3200	700

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 98
Maximum effective # of streams without language model (greedy generation): 110

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	12.8	12.6	13	13	24.8	1
n-gram	8	26	25.3	26.2	28	53	7.98
n-gram	16	37	35	43	44	87	15.94
n-gram	32	60	64	68.3	70	160	31.8
n-gram	48	92.5	96	99.4	101	270	47.5
none	1	11.9	11.7	11.9	12	18.2	1
none	8	26	25.3	30	30	46	7.98
none	16	34	32	40	41	74	15.94
none	32	55.4	61	65	66	140	31.8
none	48	84	90	93	94.2	240	47.6
none	64	107	111	115	117	320	63.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 450

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14.7	15.5	22	23	1
64	130	126	140	200	320	63.3
128	203	200	225	330	560	125.6
256	354	360	420	640	1400	246.6

Language model: n-gram

Speaker Diarization	Language model	# of streams	Throughput (RTFX)
False	n-gram	32	1200
False	none	32	1200
True	n-gram	32	100
True	none	32	100

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 93

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	13	13.6	14	16	0.999
8	27	26	30	30	34	7.98
16	36	36	43	44	46	15.95
32	59	65	72	73	76	31.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 75

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.7	13.4	13.6	20.7	1
8	30	30	33	34	46	7.99
16	42	40	50	51	66	15.97
32	74	81	84	86	118	31.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 104

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.2	12	12.7	13	15	0.998
8	25.1	24.8	26	30	33	7.97
16	33	32	40.4	41.4	43	15.94
32	57.4	64	67	68	70	31.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 98

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.4	13.5	14	14.1	16	1
8	27	26.5	28.4	30	33	7.99
16	36	35.6	40	44	46	15.97
32	60	67	74	75	80	31.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 100

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.1	12.1	12.3	12.5	15	1
8	26	25.8	30	30	30	7.99
16	36	34.4	42.5	43	46	15.97
32	60	67	70	71	80	31.9
48	88	96	100	101	110	47.8

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 70

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.8	14.6	15	15.2	20	0.999
8	34	35	38	40	50	7.97
16	50	50	60	60	80	15.9
32	80	87	100	100	140	31.6
48	120	130	140	150	250	47.2

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 104

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.2	12.2	12.7	13	15	1
8	27	27	30	30	32	7.99
16	34	33	41	42	43	15.97
32	56	62	66	67	74	31.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 40

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	19.5	21.5	21.8	34	0.997
8	43	42.7	46	50	70	7.96
16	69	69	77	80	104	15.87
32	125	130	142	146	170	31.6
48	3000	3000	5000	6300	6500	41

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 188

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	21.6	21	31.3	31.4	31.4	0.998
64	240	250	300	300	400	62.5
128	500	490	600	700	900	121
256	5000	5000	7000	8000	8000	190

Language model: n-gram

# of streams	Throughput (RTFX)
32	700

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 110

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.1	12	12.6	12.8	15.6	1
8	25.7	25	30	30	32	7.99
16	35	35	41	41.3	45	15.97
32	54	60	64	65	70	31.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 480

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.9	14	15	16	1
64	110	120	134	136	150	63.7
128	190	200	220	230	250	126.8
256	330	370	410	420	440	252

Language model: n-gram

# of streams	Throughput (RTFX)
32	1400

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 66

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.63	13.9	17.5	19.6	19.84	0.997
8	32	30	38	43	45	7.94
16	47.8	48.8	60	80	83	15.8
32	87	85	160	170	172	31.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 330

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14	18	18	18	0.997
64	150	160	179	190	190	62.3
128	260	280	320	330	330	122
256	470	520	580	600	600	235

Language model: n-gram

# of streams	Throughput (RTFX)
32	1000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 70

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	13.7	16.7	17	18	0.996
8	29	28.8	32	34	37	7.94
16	42	40	50	54	60	15.8
32	80	80	130	140	160	31

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 98

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.2	12	12.7	14	15	0.998
8	25.3	25	26	28	30	7.98
16	35.6	34.4	41.8	42.5	46	15.93
32	56	63	67	68	74	31.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 99

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.1	11.9	12.4	15	18	1
8	26	25	26	31	52	7.98
16	35.7	33	41.6	44	82	15.94
32	56	60	64.4	80	154	31.8
48	90	90	95	123	297	47.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 12

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	29	28	28.5	33	52.7	0.998
4	53.6	52	52.6	60	107	3.986
8	104.5	100	101.6	122.4	230	7.94

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 61

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	29.3	30	37.5	60	0.999
4	55	53.4	55	70	75	3.99
8	104	102	104	123	136	7.97

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	142
False	4	190
False	8	196

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 180

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10	9.8	10.1	10.3	15.6	1
8	17.44	17	18.5	19	37	7.99
16	28	27	30	30	60	15.95
32	42	40	44	46	108	31.8
48	53	50	60	60	150	47.6
64	69	68	73	75	210	63.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 180

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.84	9.9	10.22	10.29	12	1
8	16	16	17	18	20	8
16	26.2	26.5	29	29	30	15.98
32	40	40	45	47	50	31.95
48	52	52	56	58	60	47.9
64	63	63	68.6	70	74	63.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 108

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.6	10.6	11	11.4	13	0.999
8	22	22	23	24	26	7.98
16	36	40	40	40	42	15.95
32	60	60	62	63	66	31.8
48	82	82	85.3	86.6	100	47.6
64	103	101	110	120	180	63.2

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 140

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.3	11.3	11.7	11.8	14	1
8	20.4	20	22	23	25	8
16	31.6	32.7	34	35	37	15.98
32	50	50	53	54.5	57	31.94
48	66	70	78	80	83	47.9
64	81	90	95	97	100	63.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 121

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.6	10.6	10.8	10.9	12.8	1
8	20.5	20.3	21	22	25	8
16	30	30	36	36	37	15.98
32	55.5	56.1	57.5	58	62	31.9
48	76	78	80.6	82	86.7	47.85
64	97	98	102	104	120	63.7

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 110

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	12.7	18	30	37	1
8	30	26	35	50	75	7.99
16	38	36	50	64	89	15.97
32	65	62	80	90	120	31.9
48	79	77	90	104	144	47.85
64	100	100	120	140	170	63.7

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 110

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	11	11.2	11.4	12	1
8	23	22.6	25	25	27	7.99
16	35	35	39	40	41	15.98
32	59	59	61	62	72	31.9
48	83	83	90	90	130	47.8
64	104	104	110	110	170	63.6

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 95

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	10.8	11.5	12	13	0.997
8	22	21	24	25	26	7.95
16	40	40	50	50	60	15.83
32	74	72	90	100	100	31.3
48	105	98	147	150	160	46.4
64	141	126	210	250	280	61.1

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 111

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.1	11	11.7	12	14	0.999
8	23	22.5	25	26	29	7.98
16	35	35	39	39.6	44	15.93
32	58	59	61	63	67	31.8
48	80	81	84	86	90	47.6
64	100	100	105	106	120	63.3

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 103

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	10.7	11.22	15.6	16.4	1
8	24	23	25	30	48	7.99
16	36	34	38.5	50	80	15.95
32	61	57	60.4	90	150	31.8
48	87	79.7	86	130	278	47.55
64	115	100.3	122.4	264	447	63.2

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 100

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.6	13.6	14.6	21	1
8	20	20	22	24	40	7.99
16	27	27	30	32	56	15.96
32	46	45	50	53	96	31.85
48	61	58	64	67	180	47.7
64	78	72	80	84	330	63.5

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 258

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	7.6	7.2	8.3	9	17	1
8	11	10	12	13	36	7.99
16	14	13	15	17	50	15.96
32	20	18	22	24	90	31.9
48	24	22	27	30	130	47.7
64	32	28	35	38	170	63.5

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 100
Maximum effective # of streams without language model (greedy generation): 111

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	18.93	18.52	18.83	19	39	0.999
n-gram	8	40	37	50	52.8	87	7.97
n-gram	16	53	50	66	67	130	15.9
n-gram	32	66	69	72.3	74	170	31.7
n-gram	48	91	94	100	103.6	280	47.4
none	1	18	17.6	17.7	17.85	33.6	0.999
none	8	38	36	40	40	82	7.97
none	16	48	44	62	64	120	15.9
none	32	60.5	66	69.4	70	160	31.73
none	48	84.7	89	93	96	280	47.4
none	64	108	109	117	122	390	63

Chunk size (ms): 800
Maximum effective # of streams with n-gram language model: 424
Maximum effective # of streams without language model (greedy generation): 455

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	21	20.4	21	31.3	32	0.999
n-gram	64	140	130	170	220	400	63.1
n-gram	128	210	200	230	390	740	124.8
n-gram	256	374	360	420	770	2100	243
none	1	18.2	17.6	18	25	26	1
none	64	120	110	150	230	400	63.1
none	128	190	190	210	378	720	124.9
none	256	336	310	390	740	2060	244
none	384	550	500	580	1490	4640	357

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 97

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19.13	19.14	19.7	19.9	22	0.999
8	40	40	40	50	53	7.97
16	54	53	67	70	72	15.93
32	66	73.4	80	80	83	31.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 460

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	20	21	22	22	0.998
64	120	120	140	150	160	63.3
128	200	210	230	240	250	125.5
256	340	370	410	420	500	248

Language model: n-gram

# of streams	Throughput (RTFX)
32	1000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 103

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	17.8	18.3	18.5	27	1
8	37	35.8	37.3	45	80	7.99
16	50	50	64	66	109	15.97
32	64	68	71	80	180	31.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 450

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19.5	19	20	28.7	29	1
64	130	120	166	240	400	63.7
128	200	195	220	400	730	127
256	434	350	850	1500	2200	247.6

Language model: n-gram

# of streams	Throughput (RTFX)
32	640

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 94

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19.1	19.1	19.6	19.7	21.4	1
8	40	40	40	40	50	7.99
16	53	50	64	67	70	15.96
32	72	78	82	83	90	31.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 480

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19.5	19.7	21	22	22	1
64	134	129	174	200	200	63.6
128	215	210	300	350	430	126.3
256	367	372	500	680	700	249

Language model: n-gram

# of streams	Throughput (RTFX)
32	1300

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 104

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18.3	18.3	18.6	18.7	22	1
8	38	37.3	40	50	53	7.99
16	51	50	65	66	68	15.96
32	60	68.4	70.7	71.4	80	31.9
48	88	94	98	99	113	47.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 480

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19	19.1	20	20.3	20	1
64	120	120	150	200	200	63.6
128	177	170	220	300	330	126.6
256	360	374	500	740	800	249.6

Language model: n-gram

# of streams	Throughput (RTFX)
32	1770

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 70

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	21.1	20.9	21.2	21.7	30	0.998
8	49	49	60	61	85	7.95
16	60	60	76	80	120	15.85
32	78	82	90	100	170	31.55
48	114	118	130	140	300	47

Chunk size (ms): 800
Language model: none
Maximum effective # of streams without language model (greedy generation): 300

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24	23.1	28.6	29	30	0.998
64	160	147	220	320	400	62.2
128	270	250	400	600	780	121.3
256	600	480	1000	1600	2300	229

Language model: none

# of streams	Throughput (RTFX)
32	560

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 111

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18.2	18.2	18.7	18.9	20.7	1
8	35.6	35.4	37	37	50	7.99
16	51	50	64.3	65.7	70	15.96
32	63	67.6	70	72	94	31.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 500

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19.2	19.3	20	21	23	1
64	120	120	150	160	170	63.6
128	174	180	213	220	250	127
256	295	333	395	406	450	252

Language model: n-gram

# of streams	Throughput (RTFX)
32	1400

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 31

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	32	31.4	33.1	33.4	45	0.997
8	61	60	70	75	97	7.95
16	90	91	100.3	102.4	126	15.84
32	250	250	340	360	400	31.2
48	8000	8400	12000	13000	14000	31

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 148

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	36	37	45.4	46	46	0.996
64	330	340	400	400	500	62
128	730	660	1200	1300	1400	120
256	9000	10000	13000	14000	14000	150

Language model: n-gram

# of streams	Throughput (RTFX)
32	500

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 100

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18.4	17.8	18.3	27	30.5	0.999
8	40	37	50	50	100	7.97
16	48	46	62.6	66	147	15.9
32	66.4	68.4	72	100	253	31.7
48	97.4	92	102	156	440	47.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 308

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	21	19	28.3	28.7	30	1
64	160	123	320	390	460	63.1
128	250	200	550	700	800	124.6
256	605	380	1480	2060	2320	242.6

Language model: n-gram

# of streams	Throughput (RTFX)
32	660

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 7

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	54	53	60	60	113	0.997
4	97	91.5	92.6	110	265	3.97
8	2700	2700	4300	4800	6000	7.06

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 35

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	70	69.9	70.4	81	100	0.997
4	105	90	134	135	135.7	3.986
8	180	177	190	200	217	7.95

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	86
False	4	105
False	8	106

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 140

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.84	13.54	13.9	14	26.3	1
8	23	23	24	30	57	7.98
16	40.5	41	42.7	43.5	102	15.9
32	60	60	65	67	170	31.7
48	64	60	68	70	210	47.5
64	80	80	88	90	280	63.1

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 580

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14	15	22.7	24	1
64	106	96	110	207	380	63.2
128	150	135	160	300	650	125.1
256	274	250	280	610	1470	244.5
384	420	370	400	1060	2850	359
512	620	500	600	1550	4700	467

Language model: n-gram

# of streams	Throughput (RTFX)
32	2500

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 150

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.4	13.4	13.7	14	15.2	1
8	24	23	30	30	31	8
16	39	40	42	42	46	15.98
32	60	60	65	67	70	31.93
48	61.3	61.5	68	70	75	47.9
64	80	80	87	89	97	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 710

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.3	14.4	15	15.6	17	1
64	90	100	104	110	110	63.8
128	120	130	150	160	170	127.3
256	220	246	270	280	290	253.5
384	320	360	420	440	470	378.6
512	420	480	550	600	700	501
768	1500	1600	2000	2400	3000	731

Language model: n-gram

# of streams	Throughput (RTFX)
32	1000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 126

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.8	13.8	14.1	14.1	16	1
8	26	25	30	30	35	7.99
16	42	43	45	45	48	15.97
32	58	58	65	67	72	31.9
48	67	70	78	80	83	47.85
64	80	87	100	100	100	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 620

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.7	14.7	15.5	16	17	1
64	90	90	130	130	160	63.7
128	150	157	210	220	280	127
256	272	283	420	480	510	251
384	385	405	600	780	900	372.6
512	550	540	1000	1130	1500	491
768	5400	6000	9000	10000	10400	636

Language model: n-gram

# of streams	Throughput (RTFX)
32	900

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 114

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.2	15.1	15.4	15.5	18	1
8	28.7	28	29.3	30	40	7.99
16	50	53	56	56.4	60	15.97
32	64	63	70	70	75	31.9
48	85	85	90	91	97	47.8
64	104	104	113	116	140	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 560

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16.1	16	16.5	17	20	1
64	100	100	116	128	175	63.7
128	170	176	240	290	300	126.7
256	325	340	420	620	680	250.3
384	450	475	600	1000	1300	371
512	660	637	1260	1500	2000	490
768	5300	5000	8300	8650	8900	598

Language model: n-gram

# of streams	Throughput (RTFX)
32	900

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 100

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17.8	16.6	21.3	32	39	1
8	39	38.3	46	57	80	7.99
16	56	55	67	77	100	15.96
32	80	80	92	110	130	31.9
48	96	94	114	125	170	47.8
64	122	121	147	160	190	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 430

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24.5	21.6	39	39.3	40	1
64	150	150	200	200	200	63.6
128	230	230	300	320	340	126.8
256	420	420	600	640	800	250
384	640	620	1000	1200	1300	367
512	2000	2000	3000	4000	5000	480
768	7000	7000	10000	10000	12000	600

Language model: n-gram

# of streams	Throughput (RTFX)
32	600

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 110

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.4	15.3	15.8	16	17	1
8	30	30	40	40	43	7.99
16	55	56	61.4	62	70	15.96
32	64.3	63.6	70.4	73	80	31.9
48	80	84	91	93	130	47.8
64	106	105.4	113	118	170	63.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 560

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	16	16.5	16.6	18	1
64	105	104	140	150	158	63.7
128	164	177	199	205	220	127
256	300	330	370	400	400	252
384	460	500	600	700	900	373.4
512	650	670	1100	1400	1600	491
768	8000	8000	11000	11000	12000	590

Language model: n-gram

# of streams	Throughput (RTFX)
32	800

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 112

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19	18.66	19.6	20.6	30	0.999
8	25	23	30	30	58	7.98
16	34	30	40	42	93	15.93
32	46.2	44	52	55	160	31.8
48	58	53	60	64	260	47.6
64	74.5	64	72	80	440	63.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 520

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23.6	22.5	29	33.7	34	1
64	120	103	216	330	420	63.1
128	180	140	329	540	700	125.1
256	320	230	680	1190	1560	245
384	600	340	1600	2200	3200	357
512	1060	450	3350	4200	6500	456

Language model: n-gram

# of streams	Throughput (RTFX)
32	1800

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 216

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10	9.4	10.4	11.3	20.8	1
8	15	14	18	20	58	7.98
16	20	18	23.3	26	97	15.93
32	29.6	25	33	37	180	31.8
48	39	32	43	50	270	47.5
64	47	36	50	90	350	63.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 730

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14	22	22.7	24	1
64	80	57	177	300	400	63.2
128	124	78	300	530	710	125
256	220	120	570	1000	1370	245.5
384	320	160	1030	1480	2110	361
512	500	202	1600	2300	3200	468
768	1150	300	3900	5000	7400	640

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 83
Maximum effective # of streams without language model (greedy generation): 85

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	14	13.74	14.1	14.2	27	1
n-gram	8	27	26	30	32	58	7.98
n-gram	16	39	40	46	46.6	100	15.93
n-gram	32	70	72	77	78	200	31.74
n-gram	48	102	106	110	118	300	47.4
none	1	12	12	12	12	19	1
none	8	30	30	30	30	50	7.98
none	16	36	34	43	44	80	15.94
none	32	65	70	74	75	175	31.75
none	48	96	102	106.3	109	292	47.5
none	64	132	133	141	160	420	63

Chunk size (ms): 800
Maximum effective # of streams with n-gram language model: 365
Maximum effective # of streams without language model (greedy generation): 376

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	15.7	15.2	16	23.5	25	1
n-gram	64	140	137	156	210	340	63.3
n-gram	128	240	240	300	403	685	125
n-gram	256	440	440	500	780	2180	244.7
none	1	13.15	12.8	13.2	17.2	18	1
none	64	132	133	140	200	340	63.3
none	128	224	220	260	380	660	125.2
none	256	430	430	496	770	2176	245
none	384	900	687	840	2400	10370	354

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 80

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.86	12.8	13.4	13.6	16	1
8	30	30	35	35	40	7.98
16	40	40	50	50	50	15.94
32	68	75	80	80	85	31.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 370

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	14.3	15	16	16	0.999
64	130	140	150	153	155	63.3
128	240	247	270	270	280	125.4
256	430	480	530	550	560	246

Language model: n-gram

# of streams	Throughput (RTFX)
32	1100

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 83

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.2	13	13.6	13.7	19.4	1
8	25	24	30	30	50	7.99
16	36	35	43	44	78	15.98
32	70	77	82	86	156	31.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 382

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.4	14	17	20.3	20.5	1
64	130	130	140	210	330	63.7
128	222	230	255	403	650	127
256	513	434	1025	1780	2560	247

Language model: n-gram

# of streams	Throughput (RTFX)
32	620

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 80

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.8	13.86	14.26	14.34	16.2	1
8	27	27	30	30	35	7.99
16	38	36	45	50	50	15.97
32	72.4	80.5	89	90	94	31.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 383

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.5	14.64	16	16.6	18	1
64	140	140	180	200	200	63.6
128	280	270	400	440	500	126
256	500	480	900	1000	1300	247

Language model: n-gram

# of streams	Throughput (RTFX)
32	1220

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 90

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.53	13.5	13.8	13.9	16.16	1
8	25	24.3	30	30	30	7.99
16	34	33	40	42	44	15.98
32	60	70	72	73	77	31.9
48	91	102	105	106.4	120	47.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 390

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.8	13.7	14.4	15	16	1
64	140	140	160	200	240	63.5
128	230	230	300	440	500	126.2
256	460	470	700	1000	1000	248

Language model: n-gram

# of streams	Throughput (RTFX)
32	1370

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 65

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.4	15.3	15.5	16	20	0.999
8	36	40	40	41	50	7.97
16	58	60	68	73	90	15.9
32	94	100	110	120	160	31.6
48	140	145	160	170	300	47.1

Chunk size (ms): 800
Language model: none
Maximum effective # of streams without language model (greedy generation): 280

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	17.7	20.2	21	21	0.999
64	200	200	200	300	360	62.4
128	280	273	400	520	650	122.5
256	680	535	1270	1700	2500	231

Language model: none

# of streams	Throughput (RTFX)
32	630

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 85

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.4	13.5	14	14.3	16.3	1
8	26	25	30	30	30	7.99
16	35	33.6	43	44	46	15.97
32	63	70	75	76	100	31.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 414

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.6	15	16	16.3	17	1
64	124	134	145	149	154	63.6
128	220	244	260	266	270	126.7
256	400	430	500	580	700	250.3

Language model: n-gram

# of streams	Throughput (RTFX)
32	1250

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 39

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24.86	24.1	26	26	38	0.997
8	48	48	51	60	80	7.95
16	82	82	94	100	120	15.86
32	165	160	210	220	280	31.5
48	4600	4600	8170	8400	8600	38

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 170

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	29	29	38	40	40	0.997
64	300	300	330	400	400	62.4
128	600	560	900	1000	1200	121
256	6700	7000	10100	10600	11000	170

Language model: n-gram

# of streams	Throughput (RTFX)
32	550

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 82

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.74	13.3	15	19.6	1
8	26	24.4	27	34	56	7.98
16	38.3	40	45	48	93	15.94
32	70	70	80	100	230	31.74
48	110	107	120	200	450	47.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 290

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	13.9	20	20.4	20.7	1
64	150	135	264	316	357	63.2
128	270	240	520	620	700	125
256	670	480	1510	2160	2350	244.3

Language model: n-gram

# of streams	Throughput (RTFX)
32	670

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 8

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	43	41.5	42.1	47	83.7	0.997
4	70	67.4	68.6	78	150	3.98
8	180	144	300	300	500	7.92

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 41

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	37	36.5	37.5	44	60	0.999
4	71.2	70.3	71.3	84	92	3.99
8	147	144	150	170	180	7.95

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	97.4
False	4	120
False	8	124

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 133

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.5	11.4	11.6	11.8	19	1
8	18	17	19	21	38	7.99
16	30	29	32	33	67	15.95
32	50	48	50	54	123	31.8
48	68	65	70	72	200	47.6
64	80	78	86	90	265	63.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 610

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.2	11.8	12.2	17	18	1
64	83	76	91	143	280	63.4
128	150	150	170	277	540	125.6
256	292	280	320	550	1300	246.5
384	433	404	460	840	2300	363
512	600	520	630	1470	4700	476

Language model: n-gram

# of streams	Throughput (RTFX)
32	2400

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 152

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.5	10.6	11	11	12.4	1
8	17	17	19	20	22	8
16	30	30	33	35	36	15.98
32	43	42	48	50	54	31.95
48	54	54	58	60	64	47.9
64	66	66	72	73	76.7	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 690

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.5	11.7	12.4	13	14	1
64	80	80	90	93	100	63.8
128	140	147	170	170	190	127.3
256	256	280	306	310	330	253.3
384	380	410	440	450	500	378
512	500	540	640	700	1000	500
768	3300	3200	6000	7000	7000	700

Language model: n-gram

# of streams	Throughput (RTFX)
32	1300

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 107

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.05	10.07	10.3	10.36	11.5	1
8	20	20	23	24	25	8
16	30	30	35	35	37	15.98
32	53	55	60	60	70	31.9
48	70	78	84	85	90	47.8
64	92	102	110	114	130	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 500

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.5	12.48	13.1	13.5	15	1
64	100	110	120	140	200	63.7
128	203	210	300	320	386	126.4
256	360	370	600	640	660	250
384	530	530	800	1000	1400	369.5
512	900	730	1800	2000	2700	486
768	7000	7000	12000	12400	13000	530

Language model: n-gram

# of streams	Throughput (RTFX)
32	900

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 90

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.1	12	12.2	12.3	14	1
8	25	26	28	29	30	7.99
16	35	30	40	44	46	15.98
32	67	66	73	75	90	31.9
48	100	98	103	106	126	47.8
64	120	124	130	140	200	63.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 408

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	12	12.54	13	14	1
64	130	130	169	190	250	63.6
128	230	235	360	400	440	126.3
256	420	440	500	860	1000	249
384	730	673	1400	1700	2000	367.4
512	4500	5000	6600	7000	8000	423
768	13500	12500	24200	25000	26000	427.5

Language model: n-gram

# of streams	Throughput (RTFX)
32	800

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 90

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.1	13	18.3	28.6	36.1	1
8	34	33	40	50	75	7.99
16	50	50	57	70	90	15.97
32	96	96	110	123	140	31.9
48	123	124	143	153	184	47.8
64	170	160	230	270	300	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 400

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	21	18.3	35.6	36	37	1
64	146	144	190	200	210	63.7
128	260	260	330	340	400	126.7
256	460	460	700	750	930	249.5
384	800	700	1500	1700	2000	367
512	3000	3000	5000	5000	6000	470
768	10000	10700	16000	17000	17000	500

Language model: n-gram

# of streams	Throughput (RTFX)
32	700

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 89

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	11	11.3	11.4	12.2	1
8	23.5	23	26	27	30	7.99
16	38	37	45	46	47	15.97
32	67.7	67	74	76	100	31.9
48	97	96	100	100	158	47.7
64	126	124	130	135	220	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 396

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	12.1	12.7	12.8	14	1
64	120	130	142	145	150	63.6
128	220	240	258	264	270	126.7
256	440	470	550	600	700	250
384	730	710	1170	1500	1700	368
512	6000	7000	8000	9000	9000	403
768	13600	13000	23000	24000	25000	408

Language model: n-gram

# of streams	Throughput (RTFX)
32	800

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 76

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.8	12.6	13.6	14.5	21	1
8	20	20	23	25	40	7.99
16	30	29	33	36	60	15.95
32	52	49.6	55	58	150	31.8
48	73	67	74	78	300	47.6
64	122	87	99	302	995	63.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 450

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	16.2	21	24	24.5	1
64	110	94.5	160	240	310	63.4
128	200	180	300	467	600	125.5
256	390	330	670	1200	1520	246
384	830	500	2180	3000	4200	358
512	3000	670	10000	13000	15600	418

Language model: n-gram

# of streams	Throughput (RTFX)
32	1500

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 210

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	7.6	7.2	8.4	9.2	16	1
8	11.4	10.5	13	15	35	7.99
16	15.3	14.3	17.5	20	56	15.96
32	23	21	26	28	95	31.86
48	29	26	32.2	35	153	47.7
64	38	33	40	46	210	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 800

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	13	18	18	20	1
64	63	49	107	190	257	63.5
128	104	76	200	367	502	126
256	190	130	390	729	1095	248
384	290	182	713	1230	1800	366
512	420	244	1220	1850	2400	480
768	945	370	3080	4000	6040	675

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 210
Maximum effective # of streams without language model (greedy generation): 232

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	13	12.7	13	13.5	23	1
n-gram	8	23	22	25	25	47	7.98
n-gram	16	26	24.9	27	34	60	15.96
n-gram	32	42	40	50	50	110	31.83
n-gram	48	54	57	61.5	64	132	47.7
none	1	12.2	12.05	12.6	12.8	17	1
none	8	21.5	21	23	24	38	7.99
none	16	26	24	33	35	50	15.96
none	32	38	36	46.7	47.6	90	31.85
none	48	48.3	50	58.6	59.7	116	47.7
none	64	60	65	68.6	70	149	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 850

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.6	14.2	15	21	23	1
64	71	71	76.4	120	214	63.5
128	110	104	119	210	400	126.2
256	184	170	210	405	784	249

Language model: n-gram

Speaker Diarization	Language model	# of streams	Throughput (RTFX)
False	n-gram	32	1830
False	none	32	1900
True	n-gram	32	140
True	none	32	140

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 194

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.3	12.2	12.8	13	15	1
8	22.6	22.4	24	26	30	7.99
16	30	30	37	38	39	15.96
32	43	43	50	50	55	31.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 187

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.3	13.2	14	14.2	19	1
8	23	22.3	24	29	35	8
16	30	30	37	38	50	15.98
32	43	41	53	54	80	31.95

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 220

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	12	13	14	16	0.998
8	22	21	25	27	30	7.98
16	27	25	33	35	40	15.94
32	40	40	46	47	53	31.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 194

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	13	13.7	14	16	1
8	24	22.8	30	30	31	7.99
16	29	27.4	36.3	37.3	38.5	15.98
32	43	43	52	53	55	31.94

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 226

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.6	12.6	13	13	15	1
8	23	21.6	27	29	30	8
16	27	25.3	34	35	36	15.98
32	40	40	47	48	50	31.94
48	54	56.5	59	59	65	47.9

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 110

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.8	14.6	15.2	15.5	19	0.999
8	28	27	30	34	40	7.97
16	39	40	47	48	67	15.9
32	60	60.4	70	77	120	31.7
48	80	80	100	100	150	47.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 230

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.5	12.4	13	13.3	15	1
8	22	21	27	28	30	7.99
16	27	25	34	35	36	15.98
32	40	40	46	46.7	50	31.94

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 75

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	19	21	22	80	0.997
8	32	31	34	40	50	7.97
16	49	48	57	60	75	15.9
32	85	85	100	105	130	31.7
48	130	130	150	160	200	47.3
64	141	145	167	180	230	62.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 340

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	23	34.7	80	80	0.997
64	150	145	200	200	200	63
128	250	260	300	340	360	125
256	500	500	700	800	800	244
384	1000	1000	2000	2000	3000	344
512	4100	4200	6000	7000	7300	390

Language model: n-gram

# of streams	Throughput (RTFX)
32	1000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 230

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.3	12.3	12.7	12.9	15	1
8	22	21.3	25	28	29	7.99
16	27	25	34	34	35	15.97
32	38	37	46	47	50	31.93

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1040

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.7	12.6	13	15	16	1
64	62	68	72	74	82	63.8
128	94	96	115	117	200	127.4
256	150	160	200	205	220	254

Language model: n-gram

# of streams	Throughput (RTFX)
32	2390

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 150

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	13.3	16	19	19.4	0.998
8	25	24	30	32.7	40	7.96
16	32	29.4	41	50	50	15.9
32	50	50	70	83	90	31.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 630

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14	17	17	17	0.997
64	84	88	100	100	110	63
128	130	138	150	156	160	125
256	230	250	280	300	300	245

Language model: n-gram

# of streams	Throughput (RTFX)
32	1600

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 134

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	13.3	16	16	20	0.996
8	27	26	30	30	30	7.93
16	35	33	43	44	48	15.82
32	60	60	70	80	90	31.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 220

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.7	12.7	13.3	14	16	0.998
8	23	21.8	27	30	30	7.98
16	26	25	30	35	38	15.94
32	40	40	48.6	49.3	54	31.84

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 217

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	11.7	12.3	14	17.3	1
8	22	21	26	30	43	7.99
16	26	23.7	30	35	63	15.96
32	40	37	46.6	50	111	31.85
48	50	50	59	70	160	47.7

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 20

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	31	29.9	31	35	55	0.998
4	38	37	37.7	44	72.5	3.99
8	69.4	67.3	68.3	78	135	7.96

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 98

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	33	31.36	32	38	80	0.999
4	39.2	38.6	39.5	47.5	52.7	3.99
8	70	69.1	70	82	92	7.98

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	177
False	4	290
False	8	320

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 320

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.4	9.3	9.7	9.8	13	1
8	13	12.8	14	15	24.3	7.99
16	17.6	17	19.5	20.5	39	15.97
32	25	24	27	28	65	31.9
48	32	31	34	35	90	47.7
64	40	38	41	41.7	120	63.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 360

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.5	9.5	10	10	12	1
8	13	13	16	16.4	17.5	8
16	18	18	20	21	22	16
32	25	25	27	28	29.4	32
48	30	30	34	34	36	47.9
64	37	37	40	41	43	63.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 217

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.3	10.2	11	11	12	0.999
8	18.6	18	22	23	24	7.98
16	22	20	28	29	30	15.95
32	38	38.5	41	42	45	31.87
48	49	50	53	54	57	47.7
64	60	62	68	70	71	63.6

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 280

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.7	9.6	10.1	10.4	12	1
8	15.3	15	17	18	19	8
16	20	20	23.3	23.7	24.6	16
32	32	32	35	36	38	31.95
48	40	41	45	46	48	47.9
64	48	49	54	55	56.4	63.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 225

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	11	12	12	13	1
8	18	17.7	20	20	23	8
16	23	20	28.2	28.7	30	15.98
32	37	38	40.1	41	44	31.95
48	47	49	54	55	58	47.9
64	60	61	68	70	72	63.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.6	12.3	17	27.6	38	1
8	21.6	19.7	25.3	38	60	8
16	30	28	35	55	71	15.98
32	47	45	54	66	100	31.94
48	64	63	73	90	130	47.9
64	78.6	77	91	114	150	63.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 230

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	11	11.5	11.7	12.2	1
8	18	18	20	20	24	8
16	23	21	29	29.4	30.3	15.98
32	37.4	39	41.4	42	43.3	31.95
48	48.6	50	53	54	58	47.9
64	61	61.2	66.5	69	80	63.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 180

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.6	10.4	11	12	13	0.997
8	18.27	18	20	20	20	7.96
16	24	22	30	30	30	15.9
32	41	42	48	50	54	31.6
48	55	53	73	76	90	47.1
64	71	67	100	106	110	62.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 216

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.25	11.2	12	13	14	0.999
8	18	17.5	20	22	30	7.98
16	26	25	30	32	36	15.94
32	38	38.6	42	44	47	31.86
48	49.3	50.4	54	55	60	47.7
64	60	60	70	70	74	63.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 201

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.5	10	11.3	14	15.5	1
8	19	18	20	25	38	7.99
16	23	20	28	34	60	15.96
32	40	38	41.7	60	113	31.85
48	53	49	54	85	162	47.7
64	65	60	66	105	250	63.4

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 290

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.7	9.4	10.2	11.3	16.4	1
8	12	11	14	14.4	26	7.99
16	14	13	16	17	38	15.97
32	20	19	22	23.3	62	31.9
48	24	22.65	26	28	83	47.8
64	28	26	30	32	120	63.6

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 340

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	7.8	7.5	8.4	9.4	15	1
8	9	9	10.5	12	28	7.99
16	12	11	14	15	44.4	15.97
32	16	14.5	17.4	19	71	31.9
48	18	16	20	22	100	47.8
64	23	20	24	26	139	63.6

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 100
Maximum effective # of streams without language model (greedy generation): 103

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	19	19	19.4	19.65	38	0.999
n-gram	8	40	40	50	50	92	7.97
n-gram	16	50	46.6	60	66	110	15.92
n-gram	32	80	80	94	95	250	31.7
n-gram	48	112	120	126	130	320	47.4
none	1	17	17	17	17	30	1
none	8	38	37	37.6	37.9	77	7.98
none	16	50	50	60	60	100	15.93
none	32	74	80	90	90	210	31.7
none	48	105	116	119.5	122	310	47.4
none	64	138	140	150	157	430	63

Chunk size (ms): 800
Maximum effective # of streams with n-gram language model: 405
Maximum effective # of streams without language model (greedy generation): 430

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	20	20	20.7	29	29.8	1
n-gram	64	146	148	162	220	370	63.2
n-gram	128	240	250	270	410	710	125
n-gram	256	415	400	470	770	2130	244.4
none	1	17	17	18	22	23	1
none	64	133	138	150	220	370	63.2
none	128	220	210	262	400	690	125
none	256	380	360	450	750	2060	244.7
none	384	620	570	660	1570	5000	358

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 89

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	18	19	19	21	0.999
8	41	41	42.4	44	60	7.98
16	53	51	70	72	75	15.93
32	80	90	98	100	104	31.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 410

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19	19	20	20	20	0.998
64	148	160	172	180	180	63.2
128	250	270	300	310	320	125.3
256	410	450	495	510	540	246.6

Language model: n-gram

# of streams	Throughput (RTFX)
32	900

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 101

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	17	18	18	24	1
8	40	39	40	40	68	7.99
16	50	47	60	65	100	15.97
32	76	80	92	94	220	31.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 420

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	18	20	25	26	1
64	140	143	155	230	370	63.7
128	237	240	260	420	696	127
256	480	390	970	1700	2400	247

Language model: n-gram

# of streams	Throughput (RTFX)
32	570

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 90

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18.77	18.83	19.3	19.4	21.8	1
8	40.7	40.2	42.4	43	54	7.99
16	54	50	70	70	74	15.97
32	77	87	95	96	100	31.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 408

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20.2	20.4	22	22.6	23	1
64	153	160	195	200	250	63.5
128	268	267	400	420	468	126
256	480	460	800	900	1000	247

Language model: n-gram

# of streams	Throughput (RTFX)
32	1100

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 104

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	18	18	18	20	1
8	39.5	39.1	40	42	55	7.99
16	50	50	70	70	70	15.96
32	75	87	93	94	100	31.9
48	105	119	122.9	124.4	150	47.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 460

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19	19	19	20.6	22	1
64	140	147	178.5	194	234	63.6
128	235	240	310	400	430	126.3
256	380	400	500	830	967	249

Language model: n-gram

# of streams	Throughput (RTFX)
32	1530

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 60

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19	19	19	20	25	0.998
8	60	60	63	65	80	7.95
16	63	62	79	82	100	15.87
32	107	111	122	130	200	31.55
48	155	162	180	190	318	47

Chunk size (ms): 800
Language model: none
Maximum effective # of streams without language model (greedy generation): 210

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22	22	24	25	26	0.998
64	200	200	300	400	420	62
128	400	380	600	800	1000	121
256	1600	2000	2000	3000	4000	215

Language model: none

# of streams	Throughput (RTFX)
32	590

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 107

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	18	18.4	18.6	21	1
8	38.2	38	38.8	40	48.3	7.99
16	50	50	66	68	72	15.95
32	73	80	89	90	130	31.85

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 480

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	19	19.5	20.7	21	1
64	128	140	150	155	160	63.6
128	210	230	250	250	260	126.7
256	330	360	440	480	520	251

Language model: n-gram

# of streams	Throughput (RTFX)
32	1230

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 38

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25.6	24.7	27	27.5	44	0.997
8	61	60	64.6	66	102	7.93
16	87	88	93	98	134	15.83
32	180	170	240	250	300	31.4
48	4200	4200	7900	8100	8300	38

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 164

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	29	28.9	42.3	45	45	0.997
64	300	310	360	390	400	62
128	600	600	900	1000	1200	120
256	7000	6000	9000	10000	11000	175

Language model: n-gram

# of streams	Throughput (RTFX)
32	500

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 99

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18	17	18	24	28	1
8	40	37	40	50	80	7.97
16	50	45	60	65	120	15.92
32	80	86	90	110	270	31.7
48	118	116.6	127.6	220	477	47.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 330

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	18	25	25	25	1
64	158	140	285	340	383	63.2
128	280	240	533	644	730	124.9
256	610	427	1420	1900	2240	244.4

Language model: n-gram

# of streams	Throughput (RTFX)
32	653

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 8

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	50	42.8	60	70	100	0.997
4	81.7	78.6	80.4	88.6	161.6	3.98
8	200	155	330	400	635	7.92

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 41

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	49	47.8	48.5	57	70	0.998
4	80.4	79	80.8	94	102	3.99
8	156	152	160	180	200	7.95

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	100
False	4	130
False	8	132

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 148

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.6	13	13	22	1
8	20.7	20	21.2	21.6	46.6	7.98
16	30	28	33	35	70	15.94
32	48	46	50	50	130	31.8
48	65	62	70	70	200	47.5
64	77	73	80	80	270	63.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 600

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.7	13	13.6	19	20	1
64	90	80	90	156	300	63.3
128	155	140	150	290	580	125.4
256	280	250	290	560	1300	246
384	416	370	423	900	2310	362
512	570	490	590	1450	4300	473

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 158

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.4	12.5	13	13	14.3	1
8	20	19.7	21	21.5	25	8
16	29	28.4	34	34.5	37	15.98
32	48.7	49	50.7	51.4	56	31.94
48	66	66	70	72	76	47.9
64	71	72	77	78	83	63.8

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 736

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	13	13.8	14.3	16	1
64	81	81	88	90	96	63.8
128	134	135	149	153	160	127.4
256	240	243	270	280	300	253.7
384	330	350	400	410	440	379
512	450	480	550	600	800	500
768	1800	2000	3000	3000	3000	725

Language model: n-gram

# of streams	Throughput (RTFX)
32	900

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 125

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.2	13.23	13.5	13.6	14.7	1
8	24.5	24.3	25.6	26	32	8
16	33	30	44	45	50	15.98
32	64	64	70	74	80	31.9
48	80	82	92	93	96	47.8
64	94	100	112	113	120	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 550

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.4	13.4	14	14.6	16	1
64	100	105	118	124	180	63.7
128	182	183	256	280	330	126.7
256	320	330	500	570	600	250.3
384	470	470	750	960	1330	371
512	780	640	1600	2000	2400	488
768	10000	10000	15300	16000	16000	565

Language model: n-gram

# of streams	Throughput (RTFX)
32	700

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 107

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.4	14	15	15	17	1
8	30.4	30.1	31.2	31.7	43	7.99
16	50	50	58	58	60	15.97
32	80	80	80	83	90	31.9
48	98	98.5	102	104	110	47.8
64	123	122	127	130	166	63.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 480

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	14	14.36	15	16	1
64	128	125	140	158	190	63.7
128	200	203	240	330	350	126.5
256	347	340	400	730	800	250
384	550	550	1000	1200	1660	369
512	1300	900	2700	3000	3000	486
768	16000	17000	23000	24000	25000	500

Language model: n-gram

# of streams	Throughput (RTFX)
32	700

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 80

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16.6	15	20.4	32	41	1
8	33.3	31	39	50	70	7.99
16	50	45	60	80	113	15.97
32	90	90	100	120	160	31.9
48	122	120	137	160	210	47.8
64	150	150	180	200	280	63.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 350

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23.4	21	36	38	38.5	1
64	170	160	230	240	260	63.6
128	315	310	420	450	550	126.4
256	640	600	1010	1200	1300	246.5
384	2300	2300	3500	4000	5000	357
512	7000	7000	10000	10000	12000	400
768	13800	15000	20000	22000	23000	470

Language model: n-gram

# of streams	Throughput (RTFX)
32	500

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 110

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	14	15	15	16	1
8	29.7	29.4	31	31.2	40	7.99
16	40	36	56	57	60	15.97
32	74	73	80	81	100	31.9
48	97	97	102	104	145	47.7
64	120	119	125	127	200	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 490

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	15	15.6	16	18	1
64	125	125	132	135	150	63.7
128	185	200	220	220	230	127
256	310	330	396	410	460	251.7
384	470	540	700	800	870	372.5
512	800	747	1400	2000	2000	489
768	14000	15000	20000	20000	21000	510

Language model: n-gram

# of streams	Throughput (RTFX)
32	600

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 75

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22	22	23	24	31.2	1
8	26	25	26.6	29	50	7.98
16	36	36	42.7	45	70	15.95
32	59	57	63	67	171	31.8
48	83	74	82	91	400	47.6
64	170	96	200	700	1500	63.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 420

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	27	26	33	35	50	1
64	120	104	180	270	340	63.3
128	220	187	326	508	650	125.3
256	440	356	720	1260	1800	245.6
384	920	520	2500	3500	4700	355
512	3500	710	13000	15000	17000	404

Language model: n-gram

# of streams	Throughput (RTFX)
32	1360

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 206

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9	8.6	9.8	10.5	18	1
8	12	11	13	15	43	7.98
16	16	15	19	21	67	15.95
32	22	20	25	28	114	31.83
48	29.5	27	32.5	37	167	47.7
64	38.5	34	42	50	240	63.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 780

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.5	13.7	20	20.4	22	1
64	66	50	120	210	280	63.4
128	108	76	214	393	537	125.8
256	190	120	397	750	1100	247.4
384	280	160	710	1210	1750	365
512	400	210	1180	1780	2340	479
768	820	310	2680	3560	5140	675

Language model: n-gram

# of streams	Throughput (RTFX)
32	1500

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 39
Maximum effective # of streams without language model (greedy generation): 39

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	34.4	33.5	34	34	79	0.998
n-gram	8	56	56.5	62	65	150	7.95
n-gram	16	63	60	70	70	170	15.88
n-gram	32	146	132.2	170	220	470	31.5
none	1	32.8	32.2	32.4	32.4	70	0.998
none	8	54	55	60.7	63	135	7.96
none	16	60	57	61	70	150	15.9
none	32	140	126	160	200	460	31.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 173

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	29.5	28.4	29	48	49	0.998
64	240	250	282	420	700	62.5
128	520	500	568	900	2360	122

Language model: n-gram

Speaker Diarization	Language model	# of streams	Throughput (RTFX)
False	n-gram	32	430
False	none	32	430
True	n-gram	32	90
True	none	32	90

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 39

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	33.6	33.6	34.2	34.4	36	0.998
8	58	57	64	65	70	7.96
16	63	63	66	70	80	15.93

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 32

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	36	35.6	36.3	36.6	54	0.999
8	56	56	63	63	95	7.99
16	76	75	77	78	126	15.96

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 40

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	33	32.8	33.5	34	36.5	0.996
8	57	56	62	63	66	7.95
16	59	58	62	65	73	15.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 39

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	34.7	34.7	35.3	35.6	37.5	0.999
8	58.7	58.7	64	65	67	7.98
16	62.1	62	65.1	66	80	15.96

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 40

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	33	33	33.2	33.5	36.6	0.999
8	57.4	58	62.5	63.4	66	7.98
16	60	59.3	63	64	74	15.96
32	125	130	143	145	220	31.8

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 36

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	33.1	33	33.5	33.8	36	0.998
8	65	65	70	70	80	7.96
16	72	73	83	86	100	15.92

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 41

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	33	33.1	33.5	34	35.7	0.999
8	57	57	63	65	74	7.98
16	59	58	62	70	80	15.95

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 16

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	53	51.5	57.4	58.3	73	0.995
8	84	82.5	86	100	130	7.93
16	160	156	170	230	300	15.75
32	13000	14000	19000	20000	21000	16
48	22000	26000	31000	34000	34100	16.3
64	30000	34000	43000	46000	47000	16.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 67

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	48	48	61.3	61.5	61.5	0.995
64	800	700	1300	1400	1500	59.6
128	12000	12000	17000	17000	18000	69
256	30000	33000	39000	40000	40000	70
384	40000	48000	60000	60000	60000	73
512	50000	50000	78000	80000	80000	74

Language model: n-gram

# of streams	Throughput (RTFX)
32	234

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 41

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	26.7	26.65	26.9	27	30	0.999
8	57	57	61.5	62.4	65	7.98
16	59	58	61.4	63	80	15.96

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 196

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	32.6	32.5	33	35	35.2	0.999
64	220	251	270	277	300	63.3

Language model: n-gram

# of streams	Throughput (RTFX)
32	560

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 29

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	40.5	36.8	62.2	65	65.6	0.99
8	60	52	80	95	110	7.88
16	82	75	110	140	150	15.64

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 142

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	37.4	36.3	40	40	40	0.994
64	320	350	370	400	400	60.5

Language model: n-gram

# of streams	Throughput (RTFX)
32	400

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 34

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	28.6	28.2	31.6	32	32	0.993
8	60	55	80	80	90	7.84
16	74	70	100	110	110	15.6

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 41

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	32.8	32.7	33.3	34	36	0.997
8	57	60	60	60	70	7.95
16	60	59	64	70	70	15.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 41

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	32.5	31.8	32.4	33.8	57.8	0.998
8	56	56	61.5	63	110	7.97
16	59	57.4	61	64	120	15.94

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 3

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	63.7	61	64	70	137	0.996
4	4800	4300	8500	9000	10000	3.17
8	25400	25000	48000	49000	51000	3.176

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 15

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	100	100	130	140	200	0.995
4	186	182	200	210	240	3.97
8	376	373	376	410	440	7.89

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	44.4
False	4	48
False	8	48

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 64

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20.7	20	20.3	20.4	48	0.998
8	42	41.8	42.9	44	120	7.95
16	56	55	62	64	160	15.87
32	78	73	77	84	280	31.6
48	124	112	120	130	476	47.1
64	300	150	600	700	800	62.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 67

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	21.2	21.2	21.8	22	23.3	1
8	42.2	42.2	42.8	43	45.5	7.99
16	55	55	60	62	65	15.97
32	70.7	71	74	75.6	80	31.9
48	109	110	115	116	118	47.8
64	144	144	150	154	260	63.6

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 46

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24.75	24.7	25	25	27	0.998
8	55	55	59	59.4	70	7.96
16	56	56	60	60	70	15.9
32	120	119	124	160	200	31.5
48	800	800	1500	1700	2000	45
64	4300	4500	7700	8000	8000	45.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 59

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20.7	20.8	21	21.1	22.5	1
8	46	46	46.6	47	50	7.99
16	56	56	62	65	70	15.97
32	90	90	95	96	100	31.9
48	137	138	144	147	200	47.7
64	5000	4000	10000	12000	12000	59

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 46

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25.15	25.1	25.3	25.4	27.4	1
8	55	55.6	58.4	58.7	60	7.98
16	58	57	62	64	70	15.96
32	121	120	123	124	170	31.8
48	1000	1000	2000	2000	3000	46.7
64	15000	15000	25000	30000	30000	46

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 51

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23.7	22.3	27.6	38.2	46	1
8	55	53	60	75	103	7.98
16	70	70	80	100	128	15.96
32	120	120	130	150	184	31.85
48	210	190	300	318	340	47.6
64	8000	8000	14000	15000	16000	52

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 46

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25	25	25.3	25.4	26.4	0.999
8	56	55.5	59	59.7	100	7.98
16	59	57.5	63	70	104	15.95
32	123	121	123	125	220	31.76
48	840	890	1200	1300	1400	46.5
64	11000	12000	20000	20000	20000	46.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 42

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25.6	25.5	26.5	26.7	27	0.994
8	60	50	80	90	100	7.86
16	70	60	100	108	110	15.67
32	146	125	210	300	300	30.47
48	700	700	1200	1300	1400	41
64	2300	2500	3700	3800	4000	41.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 46

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	26.2	26	27	28	29.3	0.997
8	56	57	60	60	70	7.95
16	60	57.3	70	70	80	15.9
32	119	120	122	124	130	31.6
48	560	550	800	900	1000	46.2
64	4500	4700	8000	8200	8500	46.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 41

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	26.1	24.7	25.2	41.2	51.6	0.998
8	46	39	55	86	146	7.95
16	62	55.2	66.5	105	219	15.88
32	166	121	290	470	790	31.5
48	7800	7100	16100	17000	18300	41.6
64	18650	18500	29000	32000	35000	41.8

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 36

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	43.7	44.5	46	47	61	0.998
8	41.7	43.5	47.5	49	90	7.97
16	50	47	50	60	160	15.9
32	260	100	500	1200	2100	31.64

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 120

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.9	14.3	15.5	16.18	33	0.999
8	20	18.3	20.4	23.4	116	7.96
16	25.7	23.4	27	37	200	15.88
32	42	32.3	48	60	290	31.7
48	56	40	54	150	420	47.4

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 87
Maximum effective # of streams without language model (greedy generation): 88

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	13.5	13.25	13.63	13.8	26.5	1
n-gram	8	24	23	24	25	55	7.98
n-gram	16	33	30	40	41	90	15.94
n-gram	32	56	57	64	65	180	31.74
n-gram	48	90	100	100	110	300	47.4
none	1	12.3	12.14	12.3	12.4	19.5	1
none	8	22	20.8	23	24	44	7.98
none	16	27.1	26.16	27.2	30	63	15.95
none	32	51	48	60	61	160	31.76
none	48	86	90	96	100	287	47.45
none	64	118	119	123	140	420	63

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 370

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14.7	15.4	22.6	25	1
64	140	140	150	220	380	63.2
128	228	220	256	400	720	125
256	455	440	510	810	2280	243.7

Language model: n-gram

Speaker Diarization	Language model	# of streams	Throughput (RTFX)
False	n-gram	32	920
False	none	32	920
True	n-gram	32	90
True	none	32	90

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 83

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.5	13.4	14	14.5	17	0.999
8	26	25	30	30	33	7.98
16	40	40	50	50	50	15.95
32	68	73	80	81	83	31.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 63

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	14	14.6	14.7	21.7	1
8	27	26	28	29	43	8
16	37	40	40	46	65	15.98
32	73	80	86	87	139	31.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 87

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.9	12.7	13.4	14	17	0.998
8	22	22	24	25	30	7.98
16	33	31	40	41	45	15.93
32	54	60	64	66	75	31.77

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 82

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.7	13.7	14.2	14.4	16	1
8	28	28	30.1	31	33	7.99
16	38	38.3	45	47	50	15.97
32	70	80	85	87	90	31.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 86

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.9	13.2	13.3	17	1
8	23	23	24	27	30	7.99
16	31.5	30	40	40	43	15.97
32	60	64	70	70	74	31.9
48	95	103	108	110	140	47.8

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 60

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.6	15.4	15.8	16	20	0.999
8	38	40	44	48	59	7.96
16	53	54	64	66	90	15.9
32	89	96	104	110	170	31.6
48	131	140	150	160	300	47

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 90

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.77	12.8	13.3	13.5	15.6	1
8	23	22.5	24	26	30	7.99
16	33.6	31.8	41	42	45	15.97
32	58	64	67	68	90	31.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 37

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	26.7	26	27.8	27.9	39.7	0.997
8	45.9	45.3	48	60	75	7.95
16	76	76	90	94	110	15.86
32	170	160	200	230	300	31.4
48	5400	5500	9000	9000	9600	36

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30	30	40	40	40	0.997
64	300	310	359	380	400	62.2
128	660	600	1000	1100	1200	120
256	8000	9000	12000	12000	13000	160

Language model: n-gram

# of streams	Throughput (RTFX)
32	490

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 89

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.9	12.8	13.3	13.4	16	1
8	23	22	30	30	30	7.99
16	31	29	39	39.6	41.6	15.97
32	56	63	66.3	67	70	31.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 420

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.7	13.6	15	16	16	1
64	120	130	140	146	154	63.7
128	210	220	250	260	300	126.6
256	364	413	467	477	500	251.4

Language model: n-gram

# of streams	Throughput (RTFX)
32	1300

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 56

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.7	14.8	18.5	22	22.2	0.997
8	29	27	35	39	40	7.95
16	43	40	50	70	80	15.84
32	89	86	150	180	200	31.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 270

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	16	20	20	20	0.997
64	180	190	205	210	210	62
128	340	350	400	400	400	121.3
256	580	630	720	740	800	231

Language model: n-gram

# of streams	Throughput (RTFX)
32	790

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 66

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.3	14	17	17	18	0.996
8	30	30	30	40	40	7.93
16	50.5	51.5	60	70	74	15.75
32	87	90	130	140	150	31

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 90

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.9	13.6	15	16	0.998
8	23.6	23	26	28	31.4	7.97
16	34	34	40.7	41.4	45	15.93
32	57	62.2	65	67	74	31.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 82

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.9	12.63	13.16	15	19.6	1
8	23	22	26	30	56	7.98
16	34	35	39.4	43	96	15.94
32	60.5	62.6	66.4	87	233	31.74
48	102	98	116	170	447	47.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 7

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	35.3	34	34.6	40	70	0.998
4	74	71	72	84	160	3.98
8	600	500	1000	1500	2000	7.85

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 39

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	37	35.6	36.2	43.6	70	0.999
4	73	71.9	73	87	100	3.99
8	154	150	160	180	189	7.95

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	100
False	4	120
False	8	125

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 121

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.25	11.07	11.3	11.46	19.3	1
8	16	16	17	20	36	7.99
16	28	28	30	30	65	15.95
32	46	44	50	50	124	31.8
48	60	60	67	70	200	47.5
64	84	79	88	90	290	63.1

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 128

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.8	10.8	11.3	11.5	12.9	1
8	16.3	16	19	20	20.85	8
16	26	26	28	29	34	16
32	43	42	50	50	55	31.95
48	60	60	66	70	70	47.9
64	70	70	80	84	90	63.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 66

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.9	11.87	12.3	12.6	14	0.999
8	20	20	22	23	27	7.98
16	38	40	45	46	46	15.94
32	67	68	72	73	76	31.76
48	112	110	126	140	170	47.3
64	200	170	300	300	370	62.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 96

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.5	11.5	11.8	11.9	13	1
8	18	17.5	22	23	24	8
16	26	25	32	32	34	15.98
32	50	50	53	53.6	56	31.94
48	78.6	79	84	86	90	47.9
64	100	100	110	114	130	63.7

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 65

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.1	12.04	12.3	12.5	14	1
8	20	20	30	30	30	7.99
16	30	27.4	38.4	39	40	15.98
32	70	70	70	70	80	31.9
48	118	120	126	130	150	47.75
64	180	172	220	300	340	63.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 80

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.6	13.2	19	30	37	1
8	33	32	38	53	76	7.99
16	50	50	56.3	70	93	15.98
32	84	80	100	110	130	31.9
48	120	120	137	150	180	47.8
64	153	150	190	220	270	63.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 70

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	12	12.3	12.5	14	1
8	25	20	30	30	32	7.99
16	40	42	45	45	47	15.97
32	72	72	77	79.6	100	31.86
48	110	110	125	130	200	47.7
64	134	127	170	250	400	63.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 64

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.4	12.3	13.2	13.5	15	0.997
8	21	20.5	23	24	28	7.96
16	30	28	40	39	40	15.87
32	71	67	90	102	110	31.24
48	130	114	180	200	240	46
64	230	180	360	400	500	60.1

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 69

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.1	12	12.45	13	15	0.999
8	21	20	23	25	28	7.98
16	40	40	44	44	49	15.93
32	65	65	70	73	80	31.8
48	108	108	117	120	130	47.4
64	142	147	162	170	200	63

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 63

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.1	11.8	12.2	17.6	18.2	1
8	23	21	25	32	51	7.98
16	40	37.5	40	60	99	15.93
32	72	65	76.5	100	245	31.74
48	125	105	140	290	480	47.4
64	350	160	900	1280	1900	62.8

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 67

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16.6	16.3	17.4	18.4	25	1
8	21.4	20.6	23	24.5	44	7.98
16	34	30	38	40	67	15.95
32	59	55.7	66	70	180	31.8
48	96.4	80	92	110	600	47.6
64	430	250	940	1700	2800	63

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 230

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	7	6.7	7.84	8.7	15.1	1
8	11	10	12	12.5	37	7.99
16	14	13	16	18	60	15.96
32	21.7	19.7	23.8	26.5	110	31.84
48	27	24	30	33	166	47.7
64	34	29	36	42	244	63.4

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 196
Maximum effective # of streams without language model (greedy generation): 220

Language model	# of streams	Latency (ms)					Throughput (RTFX)
		avg	p50	p90	p95	p99
n-gram	1	11	10.9	11.3	11.5	20	1
n-gram	8	21	20.5	22	23	43	7.99
n-gram	16	28	27	33	34	64	15.96
n-gram	32	40	43	46	48	100	31.85
n-gram	48	53	55	60	60	128	47.7
none	1	10.4	10.3	10.4	10.5	15	1
none	8	19	18.8	19.4	19.6	32	7.99
none	16	22	22	23	23	41	15.97
none	32	34.7	33	41	42	81	31.9
none	48	47	50	54	54.7	103	47.7
none	64	58	61	63	65	135	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 790

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.7	12.4	12.9	19.2	20	1
64	75	72	82	130	200	63.5
128	113	105	140	200	360	126.4
256	180	180	220	370	685	250

Language model: n-gram

Speaker Diarization	Language model	# of streams	Throughput (RTFX)
False	n-gram	32	2100
False	none	32	2100
True	n-gram	32	120
True	none	32	110

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.2	11.1	11.8	11.96	14	1
8	22	22	24	26	30	7.99
16	29	29	34	35	37	15.96
32	40	42	50	50	56	31.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 156

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.3	11.2	11.7	12	17	1
8	21	20.5	23	24	34	8
16	28	30	33.7	34.4	45	15.98
32	43	47	49	50	70.4	31.95

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 190

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.7	10.5	11	12	14	0.999
8	19.14	19	20	21	24	7.98
16	25	30	30	30	33	15.94
32	35	36	42	43	47	31.85

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 170

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.3	11.3	11.9	12	13.9	1
8	22	22	24	25	30	8
16	28.4	28.5	35	36	38	15.98
32	44	47	54	55	60	31.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 220

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	10.9	11.3	11.4	13.7	1
8	18.2	17.8	20	20	24	8
16	23.5	23	28	29	31.7	15.98
32	35	39	41.15	41.9	48	31.95
48	48	51	54	55	64	47.9

Chunk size (ms): 160
Language model: none
Maximum effective # of streams without language model (greedy generation): 100

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.2	13	13.4	14	17	0.999
8	26	26.3	28	30	37	7.98
16	37	37	45	50	60	15.92
32	56	60	70	72	100	31.75
48	80	85	96	100	150	47.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 200

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.4	10.3	10.8	11	13	1
8	19.3	19.2	20	20	23	8
16	24	23	30	30.6	33	15.98
32	36	40	43	44	47	31.94

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 70

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.5	14.7	16.6	16.8	28.4	0.998
8	31	30	36	39	65	7.96
16	46	45	53	55	80	15.9
32	73	78	86	88	110	31.7
48	120	126	150	160	200	47.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 300

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19	18.9	28.2	29	29	0.998
64	170	170	200	230	300	62.9
128	300	300	400	400	600	124
256	600	560	800	900	1000	241

Language model: n-gram

# of streams	Throughput (RTFX)
32	940

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 225

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.46	10.4	10.7	10.9	13.6	1
8	18	17.4	20	20	23.6	7.99
16	23	22	28	29	32	15.98
32	35	38	40.8	42	50	31.94

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 930

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.6	11.4	13	14	15	1
64	64	70	76	77	80	63.8
128	90	93	110	120	130	127.4
256	160	175	213	220	240	253.7

Language model: n-gram

# of streams	Throughput (RTFX)
32	2600

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 130

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.6	12	16	16.4	16.7	0.998
8	22.5	21.8	27	28.6	30	7.96
16	30.6	30	37	39	40	15.9
32	52.4	52	83	88.5	93	31.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 570

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.5	13	16	16	16	0.998
64	87	90	104	107	110	63
128	160	160	190	200	200	124
256	260	280	320	400	400	243

Language model: n-gram

# of streams	Throughput (RTFX)
32	1700

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 130

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	11.8	14.7	15	16	0.996
8	22	22	25	27	30	7.94
16	31	30	37	40	43	15.87
32	50	50	75	85	90	31.44

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 200

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.6	10.6	11.1	12	13.4	0.999
8	19.6	19.4	21	23	24	7.98
16	26.5	25	31.7	32.6	40	15.94
32	38	40	46	48	60	31.84

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 195

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.6	10.3	11	13	17	1
8	20	19.5	21	25	41	7.99
16	27	27	31.5	34	62	15.96
32	40	40	43	50	104	31.86
48	50	50	55	70	145	47.7

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 18

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18.8	18.25	18.8	22	32.8	0.999
4	30.7	29.6	30.3	38	61	3.99
8	57.3	55.3	57	66	125	7.97

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 93

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	19.6	20	25.7	50	1
4	32	31	31.9	40	50	3.994
8	60	57	70	76	90	7.98

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	1	190
False	4	270
False	8	280

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 256

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	8.7	8.6	8.8	9	13.6	1
8	15	15	16	16.3	28	7.99
16	20	20	23	24	46	15.96
32	32	32	33.6	35	80	31.9
48	40	40	44	46	100	47.7
64	53	51	57	60	140	63.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 240

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	8.34	8.4	8.7	8.9	10.1	1
8	17	17	18.3	19	23	8
16	23	23.5	26	27	30	16
32	35	36	38	38	40	31.96
48	46	46	52	53.1	56	47.9
64	55	56	62	64	70	63.9

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 187

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.4	9.4	9.7	10	10	0.999
8	16.5	16.4	18	18	20	7.99
16	20	20	23	26	28	15.96
32	37	37.6	40	40	42	31.9
48	49	49	53	55	60	47.7
64	59.7	60.4	65	66	70	63.6

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 217

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.1	9.1	9.5	9.6	10.7	1
8	15.8	15.8	17	18	20	8
16	24	25	26	27	30	16
32	34	34	37	39	42	31.95
48	46	46	50	52	60	47.9
64	54	55	60	60	70	63.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 190

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.54	9.5	9.66	9.7	12	1
8	17	16.4	18	20	20	8
16	22	22	25	27	28	16
32	37	38	40	41	47	31.95
48	48	49	52	54	60	47.9
64	59.5	60	64.6	66	70	63.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.4	11	16.6	27.5	34.6	1
8	20	18	24.7	42	67	8
16	28	26	33	53	76	15.98
32	47	44	60	70	100	31.94
48	61	60	70	95	114	47.9
64	75	73	86	100	142	63.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 196

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.2	9.2	9.6	9.8	10.6	1
8	17	16.4	18	19	20	8
16	23	23	26.7	27.4	30	15.98
32	36.6	37	39.5	40.5	43	31.95
48	48	49	52	53	56	47.9
64	60	60.5	65	67	82	63.8

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 156

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.7	9.6	10	10	10	0.997
8	17.4	17	19	20	23	7.96
16	22.3	22.3	23.7	27	30	15.9
32	43	45	50	60	60	31.6
48	60	60	80	90	100	47
64	80	76	110	120	120	62.3

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 187

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.6	9.5	10	10	12	0.999
8	18	17	20	20.7	24	7.98
16	28	28	30.4	31	35	15.95
32	38	39	42	45	50	31.85
48	49.4	50.5	56	58	65	47.7
64	60	62	69.4	71.6	81	63.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 180

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	9.3	9.1	9.5	12.8	13.2	1
8	17.4	16.5	20	22.7	34	7.99
16	27.3	26.6	28.6	36	60	15.96
32	40	38	42	57	100	31.87
48	52	49	54	81	141.5	47.7
64	64	60	66	100	220	63.5

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 130

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.1	10.8	12	13	18	1
8	16.7	16	19	20	31	7.99
16	23.7	23	27	28.3	46	15.97
32	37	36	41	42.5	75	31.9
48	46	45	51	53.6	99	47.8
64	57	55	61	65	170	63.6

Chunk size (ms): 100
Language model: n-gram
Maximum effective # of streams with n-gram language model: 280

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	6.9	6.6	7.6	8.4	14	1
8	9.2	8.7	10	11	27	7.99
16	13	12.4	15	17	41	15.97
32	17.5	17	20	23	68	31.9
48	23	22	28	30	94	47.8
64	27	25	30	36	120	63.7

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	27.6	29.14	29.95	53.7	28.16	0.995

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	26.64	28.2	31.86	37.6	26.93	0.994

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	25.63	26.66	27.36	33.75	25.53	0.997

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	25.43	27	34.16	35.2	26.43	0.99

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	27.4	28.7	29.4	40.6	27.4	0.997

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	25.6	26.3	26.5	33.9	25.86	0.997

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	24.4	24.84	24.93	47.6	24.9	0.997

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	26.2	27	29.7	40.2	26.27	0.995

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	41.6	42.24	45.1	61.2	42.4	0.99

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	23.93	24.5	24.6	33.05	24.15	0.997

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	29.1	40.8	44.6	44.6	31.3	0.98

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	26.5	41.05	41.05	41.05	29	0.974

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	26.4	28.1	33.7	33.7	26.94	0.989

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	25.2	26.3	29.36	39.4	25.55	0.996

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	18.37	19.05	19.36	29.44	18.73	0.997

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	18.07	18.86	18.9	27.8	18.1	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	22.7	23.38	25.56	32.5	23.2	0.993

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	18.47	19.66	21.44	29.3	18.66	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	22.75	23.17	23.66	32.16	22.98	0.997

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	19.14	34	52.7	67.6	22.96	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	23.97	25	27.27	34.8	24.03	0.996

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	23.57	33.93	33.93	33.93	24.86	0.981

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	23.73	24.7	30.45	32.3	24.06	0.991

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	22.4	23.24	24.5	32.9	22.6	0.997

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	40.9	44.4	48.1	184	44	0.992

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	41.6	100	184.6	189.6	52.8	0.992

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	37.8	40.55	44.4	50.7	37.7	0.989

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	64.4	592	713	754	192.5	0.98

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	38.5	39.8	40.5	58.4	38.9	0.994

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	36.1	37.9	39.3	73.6	37.06	0.995

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	201.8	888	1032	1130	290	0.994

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	73.3	76	79.7	136.6	76	0.952

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	34.64	35.4	36.25	47.6	34.96	0.996

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	45.5	72.4	77.7	77.7	50.4	0.967

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	2573	2806	2806	2806	2360	0.3496

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	41.4	50.4	52.1	54.7	42.2	0.984

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	37.8	169.7	325	570	70.4	0.986

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	29.77	31.9	32.5	78.1	30.94	0.995

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	31.44	192.5	227	306	69.3	0.996

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	40.24	188.7	189.4	254.6	90.7	0.965

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	35.6	37.6	41.3	52.8	36.3	0.995

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	1790	3250	3410	3600	1598	0.769

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	718	1560	1630	1843	742	0.978

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	1007	1260	1260	1260	847	0.711

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	37.7	48.9	55.7	176.7	44.9	0.986

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	34.6	54.8	183	432	51.8	0.994

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	14.2	14.87	14.96	99	15.45	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	14.4	15.97	17	99.8	17.36	0.997

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.48	14.38	14.8	102.8	14.8	0.999

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.65	15.34	17.65	99.3	17.64	0.995

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	14.16	14.8	15	93.2	15.52	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	14.1	14.55	14.83	93.7	15.23	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	15.3	16.17	16.66	96.4	16.78	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.97	15.1	17.28	98.8	15.5	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	24.3	25.75	62	219	32.7	0.994

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.26	14.38	14.97	192.7	16.73	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	16.1	20.7	98	98	21.95	0.99

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	14.43	97	97	97	23.05	0.987

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.23	14.88	17.96	155.8	19.73	0.995

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.46	14.33	16.06	102	14.92	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	11.15	12.1	12.56	187	14.06	0.999

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	11.15	11.96	12.31	103	12.6	0.999

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.2	14.22	14.44	94	16.82	0.995

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	11.35	12.02	12.66	91.2	12.73	0.999

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.26	14.07	14.28	107.1	15.36	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.16	20.66	36.24	96.5	16.22	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	12.77	13.44	14.02	100.8	14.3	0.998

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.03	93.8	93.8	93.8	20.98	0.99

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	13.63	14.98	17.28	93.8	17.24	0.995

Chunk size (ms): 160

Language model: n-gram

# of streams	Latency (ms)					Throughput (RTFX)
	p50	p90	p95	p99	avg
1	12.98	13.47	13.72	98	14.2	0.999

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 33

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	50	49.6	50.1	50.4	100	0.997
8	70	70	80	80	130	7.96
16	100	100	111	114	230	15.87
32	300	280	450	500	660	31.4
48	12100	13000	23200	24000	24700	34.2
64	22000	24500	38400	39100	39700	34

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 150

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	52.6	52	53	64.3	67	0.998
64	300	320	340	550	700	62.7
128	580	570	680	900	2000	123.2
256	17000	19000	30000	30000	38000	154
384	31000	32600	50000	53000	62600	155.5
512	45500	47000	71000	73000	86800	159

Language model: n-gram

# of streams	Throughput (RTFX)
32	423

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 52

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22.3	21.76	22	22	42.6	0.999
8	50	48	50	60	100	7.97
16	57	56	60	63	125	15.9
32	100	95	100	100	280	31.7
48	170	145	200	330	500	47.3
64	10000	10000	19000	20000	20400	52.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 236

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24	23.4	23.8	32	34	0.999
64	185	180	190	300	370	63.2
128	360	370	386	510	740	124.8
256	1300	800	1800	2200	12000	240
384	17000	18000	27400	27800	34700	246
512	28000	32000	40500	41000	50000	250
768	45000	50000	68000	70000	80000	256

Language model: n-gram

# of streams	Throughput (RTFX)
32	1100

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 36

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	49	48.6	49	49	100	0.997
8	70	80	85	85	120	7.97
16	90	100	100	102	200	15.9
32	300	286	400	485	780	31.5
48	10000	11000	21000	22000	23000	36.6
64	20300	23600	35400	36400	37000	37

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 173

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	52	51.2	52	63.3	65	0.998
64	300	300	310	400	500	63
128	520	550	600	790	1500	124
256	14000	15000	25000	26000	35000	172
384	27000	27000	44000	48000	58000	174
512	39000	38000	61900	63000	80500	175

Language model: n-gram

# of streams	Throughput (RTFX)
32	460

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 57

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22.3	21.7	21.8	21.9	42	0.999
8	50	50	55	56	100	7.97
16	55	53.5	60	62	110	15.93
32	92	90	94	97	264	31.7
48	150	134	145	230	500	47.4
64	8000	7200	16000	17000	17700	57

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 266

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	21.4	20.9	21	28.25	30	1
64	174	171	177	240	334	63.3
128	350	340	350	460	663	125.2
256	860	700	1000	1500	6000	243.5
384	14400	14500	24000	24800	32800	267
512	24000	27000	37000	37500	47000	269
768	40000	44000	61000	64000	75000	272

Language model: n-gram

# of streams	Throughput (RTFX)
32	1200

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 32

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	49	48.9	49.3	49.4	100	0.997
8	70	70	80	80	130	7.96
16	90	90	100	100	200	15.9
32	200	200	400	500	800	31.5
48	13000	14000	25000	26000	26000	33.3
64	24000	27000	42000	43000	44000	33

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 150

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	52	51.4	52.4	63.3	65	0.998
64	300	300	320	400	500	63
128	530	560	610	810	1550	123.9
256	16000	17000	30000	30000	40000	160
384	30000	30000	50000	56000	66000	157
512	46000	42000	74000	75000	93000	155

Language model: n-gram

# of streams	Throughput (RTFX)
32	460

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 50

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22.5	22	22.2	22.3	44.4	0.999
8	50	50	50	60	97	7.97
16	55	53	60	60	110	15.93
32	96	91	98	103	260	31.7
48	160	140	160	260	460	47.4
64	9000	9000	18000	20000	20000	53

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 230

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22.7	22	23	30.3	30	1
64	180	173	183	250	350	63.2
128	353	345	360	470	690	125
256	1000	700	1200	1700	9000	242
384	19000	20000	30000	30000	37000	240
512	30000	35000	45000	46000	55000	236
768	49000	55000	73700	76200	88800	236.5

Language model: n-gram

# of streams	Throughput (RTFX)
32	1200

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 77

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.4	14.1	14.6	14.78	27	1
8	33	34	36	36	60	7.98
16	42	45	48	50	110	15.92
32	77	80	84	87	270	31.7
48	115	116	120	134	360	47.3
64	150	140	160	280	500	62.9

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 310

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	15.8	17	22.6	25	1
64	150	150	200	300	380	63.2
128	240	240	300	410	650	125.3
256	480	470	600	1000	2200	244
384	2000	1300	3000	4000	13100	344
512	13000	14400	23000	24000	30740	320

Language model: n-gram

# of streams	Throughput (RTFX)
32	870

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 122

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	10.6	10.5	10.9	11.05	16	1
8	22	20	24	24	39	7.99
16	30	30	32	33	70	15.95
32	51	50	53.6	55	130	31.8
48	65	65	70	73	200	47.6
64	83.3	82.5	90	95	300	63.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 500

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.4	11.1	12	15	17	1
64	89	86	100	150	245	63.4
128	158	156	175	280	464	126
256	303	295	330	550	1200	248
384	450	420	500	930	2260	366
512	770	580	1300	1800	6200	477
768	11000	11500	17000	17000	27000	555

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 81

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.3	15	15.7	16	29	1
8	33	30	35	35	66	7.98
16	42	45	47	47	105	15.93
32	73	77	80	83	200	31.76
48	110	110	118	120	290	47.5
64	140	135	150	190	400	63.1

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 360

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16.7	16.4	17.5	23.8	27	1
64	140	140	160	200	300	63.4
128	240	250	300	370	540	125.6
256	420	430	500	720	1500	247
384	1040	730	1030	2200	12000	357
512	9000	9000	14500	17500	30200	373

Language model: n-gram

# of streams	Throughput (RTFX)
32	1020

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 130

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.5	11.3	12	12	17	1
8	18.6	18.3	21	21	37	7.99
16	30.5	29.47	32.7	34	60	15.95
32	48.7	48.3	51.6	53	107	31.8
48	66	65	70	72	160	47.6
64	80	81	86	90	240	63.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 570

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.05	11.7	12.5	16	20	1
64	86	83	90	133	219	63.5
128	150	153	170	252	420	126
256	280	280	300	492	1100	248.7
384	404	410	450	730	2060	368
512	600	540	660	1480	4600	482
768	8000	8000	12000	13000	26000	590

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 82

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.4	15	16	16	30	1
8	35	34	36	36	70	7.98
16	42.2	45	47	47	100	15.93
32	70	75	80	80	180	31.8
48	100	110	115	120	280	47.5
64	140	137	148	190	390	63.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 370

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	17	17.7	24	27	1
64	130	140	160	200	280	63.4
128	250	250	300	360	530	125.7
256	420	440	500	705	1490	247
384	980	713	1000	2000	10400	359
512	8300	8500	13000	16000	30400	373

Language model: n-gram

# of streams	Throughput (RTFX)
32	1040

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 131

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.8	11.6	12	13	18	1
8	19	18.8	20.5	21	38	7.99
16	33	32	34	35	67	15.95
32	49	48	50	52	110	31.84
48	67	66	70	71	157	47.6
64	80	82	88	90	250	63.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 580

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	11.7	12	16.4	20	1
64	90	84	100	135	220	63.5
128	147	150	170	250	420	126
256	280	280	300	490	1000	248.7
384	406	410	450	727	2040	368
512	590	540	630	1450	4300	482
768	7500	7000	12000	13000	26000	590

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 81

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20.5	20	20.6	20.8	43	0.999
8	60	60	60	62	110	7.97
16	67	70	80	80	160	15.9
32	93	95	104	110	280	31.65
48	130	130	150	160	400	47.3
64	180	174	200	300	630	62.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 300

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23	22	23	33	34.3	0.999
64	180	170	194	400	500	62.9
128	270	240	300	600	1100	124
256	510	450	800	1000	2870	240
384	2000	1900	3000	4000	10800	336
512	11400	12500	20000	22000	25000	320

Language model: n-gram

# of streams	Throughput (RTFX)
32	740

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 119

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.8	13.3	13.5	23	1
8	28	27	29.9	30.6	50	7.98
16	37	36	40	40	90	15.93
32	50	50	60	60	134	31.8
48	67	66	76	80	240	47.5
64	86	83	94	100	294	63.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 480

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.5	14.1	14.4	20	22	1
64	94	90	100	180	300	63.3
128	166	160	190	340	600	125.4
256	310	290	335	640	1400	246
384	480	420	600	1180	3030	361.5
512	1000	660	1600	2300	7800	467
768	11000	13000	18000	18000	26000	530

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 94

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	20	20	20.6	42	0.999
8	50	54	60	60	100	7.97
16	66	70	70	75	140	15.9
32	90	100	100	110	230	31.7
48	120	127	143	146	316	47.4
64	150	149	170	180	425	63

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 390

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22	21	22	31	33	0.999
64	150	153	172	230	350	63.2
128	240	250	270	394	630	125.3
256	400	400	470	730	1570	245.7
384	710	630	750	1670	5700	360
512	5000	5400	8000	9000	21000	415

Language model: n-gram

# of streams	Throughput (RTFX)
32	950

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 136

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.5	12	12.4	12.5	23	1
8	24	23	27.46	28	50	7.98
16	36	35	37	37.8	77	15.94
32	50	50	53	54	130	31.8
48	68	65.5	69	72	190	47.5
64	85	80.4	85	90	280	63.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 570

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.4	13	13.6	19	20	1
64	90	80	90	150	280	63.4
128	145	142	160	281	530	125.6
256	270	260	287	550	1250	246.7
384	398	374.4	420	870	2260	363.6
512	590	500	600	1530	4700	475
768	5400	5000	10000	11000	22000	600

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 95

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	20	20	21	43	0.999
8	50	50	60	60	100	7.97
16	67	70	74	76	140	15.9
32	90	90	100	102	240	31.7
48	120	130	140	150	320	47.4
64	160	152	176	200	430	63

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 390

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23	23	24	33	35	0.999
64	160	160	180	250	400	63.1
128	250	257	300	400	690	125
256	430	420	500	770	2060	244.6
384	730	643	760	1700	5900	358
512	4000	4000	7000	7000	23000	413

Language model: n-gram

# of streams	Throughput (RTFX)
32	940

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 140

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.5	13	13.4	13.6	24	1
8	20.8	20.05	20.8	21.3	46	7.98
16	33	33	37	38	70	15.94
32	52	50	54	57	130	31.8
48	70	68	74	76	180	47.6
64	90	86	92	96	280	63.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 580

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.6	14	14.8	20	22	1
64	90	85	95	160	300	63.3
128	170	150	170	296	570	125.5
256	290	268	304	580	1320	246
384	420	384	450	930	2340	362.5
512	610	510	630	1540	4720	473
768	6000	6000	10000	13000	23000	590

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16.7	16.3	17	17	33	0.999
8	40	30	40	40	70	7.98
16	47.3	52	54	54	100	15.93
32	60	60	70	70	140	31.8
48	76	78	81	83	160	47.6
64	90	90	96	100	250	63.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 640

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18.3	18	19	25.3	30	1
64	110	105	128	170	270	63.4
128	146	140	190	240	420	126.2
256	233	240	290	450	850	249
384	345	360	410	658	1470	367.5
512	480	480	540	1000	2700	484

Language model: n-gram

# of streams	Throughput (RTFX)
32	1600

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 230

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	11.8	12.2	12.4	19	1
8	24	23	26	26.5	40	7.99
16	28	27.4	28.7	30	56	15.95
32	39	38	40	42	93	31.84
48	48	47	52	54	130	47.7
64	57	55	64	68	160	63.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 910

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.9	13.6	17	20	1
64	65	60	70	114	220	63.5
128	110	100	120	200	390	126.3
256	180	166	200	360	720	250
384	250	230.6	276	524	1240	369
512	323	298	358	700	1900	487
768	510	453	580	1400	3800	710

Language model: n-gram

# of streams	Throughput (RTFX)
32	4000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 140

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.8	15.45	16	16.5	33.6	0.999
8	39	39	40	40	76	7.98
16	44	47	50	50	100	15.93
32	60	61	67.4	69	225	31.7
48	74	76	81	84	230	47.5
64	90	90.7	97	100	290	63.1

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 500

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18.7	18.2	19.6	27.9	30	1
64	120	116	140	280	370	63.2
128	160	144.7	200	370	560	125.6
256	270	250	300	600	1200	247
384	420	380	700	1000	2200	363
512	700	550	1380	2000	4000	472.5

Language model: n-gram

# of streams	Throughput (RTFX)
32	1240

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 200

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.2	10.92	11.4	12	19.3	1
8	20	20	24	25	37.3	7.99
16	27	26.5	29	30.5	57	15.96
32	40	40	43	44	115	31.8
48	47	47	52.4	53.6	140	47.65
64	57	56	64	67	190	63.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 800

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.5	12	14	17	20	1
64	70	66	80	130	220	63.5
128	112	106	130	222	390	126.3
256	180	170	200	400	730	249.3
384	259	237	291	580	1300	369.5
512	340	300	380	780	2090	487
768	700	508	1200	1800	6000	708

Language model: n-gram

# of streams	Throughput (RTFX)
32	3000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 155

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.5	15.18	15.6	15.8	33	0.999
8	40	40	40	40	70	7.98
16	47	49	51	52	100	15.94
32	57	60	66.4	68	144	31.8
48	70	74	78	80	160	47.6
64	85	86.6	90.8	94	210	63.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 620

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18.2	17.8	18.8	27	30	1
64	107	110	120	170	250	63.4
128	140	130	160	240	410	126.2
256	220	230	300	450	780	249
384	339	350	400	670	1450	369
512	480	480	600	1080	2800	485

Language model: n-gram

# of streams	Throughput (RTFX)
32	1600

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 220

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11	10.74	11	11.2	18.8	1
8	20	20	22	23	40	7.99
16	26	25	27	28	50	15.96
32	39	39	42.2	43	90	31.85
48	50	50	53	55	125	47.7
64	55	54	62	64	144	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 920

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12	11.6	12.3	16.9	20	1
64	65	60	70	116	210	63.5
128	103	95	110	200	366	126.4
256	170	160	200	360	690	250
384	240	226	270	525	1200	370.6
512	320	290	360	700	1880	488
768	510	450	560	1400	3880	716

Language model: n-gram

# of streams	Throughput (RTFX)
32	3000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 156

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.5	15.17	15.6	16	33	0.999
8	38.2	39	39.9	40.2	67	7.98
16	42	47	50	50	100	15.93
32	55	57	65	67	140	31.8
48	70	72	75	77.8	150	47.7
64	84	87	90	95	230	63.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 650

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18.5	18	19.5	26.9	30	1
64	110	106	130	175	260	63.4
128	145	140	180	255	420	126.2
256	230	230	300	470	800	249
384	350	350	400	700	1500	369
512	500	480	600	1200	3000	485

Language model: n-gram

# of streams	Throughput (RTFX)
32	1600

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 220

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.1	10.9	11.3	11.63	19	1
8	20	20	20	20	35	7.99
16	28	27	30	30.4	50	15.96
32	40	39	43	44	90	31.85
48	50	50	50	54	124	47.7
64	57	56	62	65	150	63.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 900

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.4	12	12.4	17	20	1
64	64	60	70	113	210	63.5
128	110	104	130	205	370	126.3
256	180	170	200	350	680	250
384	240	230	270	510	1166	371
512	314	293	350	690	1600	489
768	506	450	570	1370	3800	716

Language model: n-gram

# of streams	Throughput (RTFX)
32	4000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 32

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	50.6	50.1	50.7	50.9	100	0.997
8	70	60	104	106	150	7.96
16	106	112	117	120	260	15.85
32	640	630	900	1000	1200	31.2
48	14400	16000	27000	27000	28000	32
64	24000	26000	42000	43000	43000	32.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 142

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	53.3	52.6	53.5	66	68	0.998
64	350	350	370	600	700	62.6
128	660	600	800	1300	2600	122
256	19000	20000	32000	33000	40500	146
384	34300	36000	57000	58000	65600	148
512	48000	50000	77000	79000	90400	151

Language model: n-gram

# of streams	Throughput (RTFX)
32	400

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 49

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22	21.5	21.9	22.1	43	0.999
8	50	50	60	60	110	7.97
16	58	56	62	70	130	15.9
32	100	100	100	110	305	31.67
48	180	150	300	400	570	47.3
64	12400	12200	22000	22700	27000	49.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 230

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24.6	24	25	32.7	35	0.999
64	200	194	206	290	410	63.1
128	370	375	403	520	800	124.7
256	3000	2000	4000	5000	13000	233
384	20000	21000	30000	30000	37000	233
512	31000	37000	44000	45000	52500	237
768	50000	56000	73000	74000	83400	246

Language model: n-gram

# of streams	Throughput (RTFX)
32	1000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 33

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	51.4	51.2	51.8	52.1	105	0.997
8	70	70	90	100	136	7.96
16	100	100	120	120	220	15.88
32	480	470	660	730	900	31.4
48	12300	13000	24300	25000	25700	33.8
64	22700	26000	39500	40400	41000	34

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 159

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	54.3	53.8	54.6	66.6	70	0.998
64	300	320	340	400	520	62.9
128	570	560	654	900	1800	123.5
256	17000	20000	30000	30000	38400	160
384	31000	33000	52000	54000	62500	160
512	43000	42000	68000	69000	86700	163.7

Language model: n-gram

# of streams	Throughput (RTFX)
32	430

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 54

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25.1	24.5	24.9	25.1	48	0.998
8	50	52	60	60	100	7.97
16	58	56	62	64	120	15.92
32	100	95	100	105	280	31.7
48	157	140	170	260	430	47.3
64	9200	8600	17900	18800	19500	54.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 248

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24.3	23.6	24	33	35	0.999
64	183	180	190	260	380	63.2
128	360	360	387	510	760	124.8
256	1100	730	1600	2000	10000	240
384	17000	17700	27000	28000	35000	250
512	28000	31500	41000	41600	50000	253
768	46000	50000	67800	69500	79700	256.5

Language model: n-gram

# of streams	Throughput (RTFX)
32	1100

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 33

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	52	51.3	52	52.4	106	0.997
8	70	80	90	90	130	7.96
16	90	90	114	120	220	15.87
32	450	400	700	700	900	31.4
48	12000	12000	23400	25000	25900	33.4
64	22300	24000	40000	41000	41800	33.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 157

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	55	54.4	56	67.7	70	0.998
64	330	335	360	430	550	62.9
128	600	600	680	1000	2300	123.3
256	16000	17000	30000	30000	39400	156
384	32000	33000	55000	55000	64000	160
512	45000	46000	72000	76000	88500	160

Language model: n-gram

# of streams	Throughput (RTFX)
32	430

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 54

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25.4	24.7	25.2	25.4	49	0.998
8	50	50	50	60	110	7.96
16	56	54	60	65	120	15.9
32	100	96	100	100	284	31.7
48	156	139	180	270	420	47.3
64	9800	9200	18600	19400	20100	54

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 243

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24.5	23.7	24.4	33	36	0.999
64	190	187	200	270	390	63.2
128	360	360	400	510	766	124.7
256	1500	900	2000	2500	12400	236
384	18000	19000	28000	29000	36000	240
512	29000	33000	42400	43200	51700	247
768	47000	50000	70000	72000	81700	250.5

Language model: n-gram

# of streams	Throughput (RTFX)
32	1100

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 85

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	20	20	21	40	0.999
8	64	64	66	67	115	7.97
16	69	73	80	80	160	15.9
32	100	102	110	120	300	31.6
48	134	130	150	160	400	47.2
64	172	163	190	260	600	62.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 320

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	23	23	24	33	34	0.999
64	180	170	204	400	500	63
128	290	270	330	600	1000	124.3
256	520	470	700	1000	2500	242
384	1260	910	2200	3000	9600	350
512	9600	10500	16800	18000	25000	340

Language model: n-gram

# of streams	Throughput (RTFX)
32	770

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 124

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.5	13.2	13.7	14	23.6	1
8	28	28	31	31	53	7.98
16	38	37	40	40	90	15.93
32	50	50	60	60	145	31.8
48	63.8	64	70	75	220	47.5
64	84	82	90	100	295	63.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 480

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14.4	15	20.6	21	1
64	94	89	100	180	310	63.3
128	162	156	180	310	570	125.5
256	310	290	340	600	1400	246.3
384	477	422	540	1100	2960	362
512	860	610	1400	2000	7000	470
768	10000	11000	16000	16600	25000	550

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 93

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22	21	22	22	43	0.999
8	60	60	60	60	100	7.97
16	70	73	77	80	150	15.9
32	90	99	102	106	240	31.7
48	128	130	146	149	330	47.4
64	154	150.8	177	190	430	63

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 390

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	24	24	25	34	35.4	0.999
64	160	160	200	247	371	63.2
128	250	260	300	390	640	125.2
256	420	420	500	740	1800	245.4
384	720	640	800	1730	6300	359
512	6000	6000	10000	12000	24300	414

Language model: n-gram

# of streams	Throughput (RTFX)
32	920

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 135

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.7	14.46	14.8	15	25	1
8	30	30	30	30	52	7.98
16	36	35	37	40	80	15.94
32	53	52	57	60	130	31.8
48	66.6	64.9	68.7	70.3	180	47.6
64	83	81.4	86	92	270	63.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 556

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.7	15.2	15.7	22.1	23	1
64	90	83	90	161	300	63.3
128	150	144	160	300	575	125.4
256	280	268	300	580	1320	246
384	420	386	440	970	2500	362
512	620	510	620	1600	5400	473
768	6500	6000	12000	13000	24000	580

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 94

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22	21	22	22	44	0.999
8	60	50	60	65	110	7.97
16	70	71	76	78	150	15.9
32	90	100	110	110	250	31.7
48	125	125	145	148	340	47.4
64	160	153	187	200	400	63

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 390

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	25	24	26	35	36.2	0.999
64	160	160	200	240	380	63.2
128	254	257	300	410	690	125
256	440	430	500	764	2110	245
384	730	630	800	1800	6400	357.6
512	5600	6000	10000	12000	24300	410

Language model: n-gram

# of streams	Throughput (RTFX)
32	900

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 133

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.8	14.5	15	15.2	26.4	1
8	25	20	30	30	52	7.98
16	34	33	36	40	80	15.94
32	53	51	55	57	133	31.8
48	67	66	70	72	200	47.5
64	83.9	82.4	90	93	280	63.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 550

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.93	15.5	16	22	24	1
64	90	83	100	160	300	63.3
128	152	145	170	300	580	125.4
256	286	270	300	580	1330	246
384	424	390	440	983	2600	362
512	630	520	640	1600	5410	472
768	7000	6000	13000	14000	24000	580

Language model: n-gram

# of streams	Throughput (RTFX)
32	2000

Below are tables which demonstrate what effect does number of CPUs have on latency and throughput. Measurements were made on-prem for English Conformer model.

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 176

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	13.6	14	14	28	1
8	26.2	25.5	26.3	26.7	61	7.98
16	37	33	45	45.7	90	15.93
32	53	54	63.5	64	150	31.8
48	63	66	71	74	190	47.6
64	75	77	84	87	260	63.3

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 180

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.96	13.63	14.05	14.3	29	1
8	26	25.4	26	26.3	60	7.98
16	35	32	44	45	88	15.93
32	50	50	62	63	140	31.8
48	60	65	69	72	170	47.6
64	74	76	81.2	84	250	63.3

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 183

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	13.7	14.1	14.4	29	1
8	26.2	25.5	26	26	63	7.98
16	34	30	44	44.7	85	15.93
32	50	50	62	63	140	31.8
48	59	60	68	72	160	47.6
64	73	76.5	81	82.7	240	63.3

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 183

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.9	13.6	14	14.2	28	1
8	26	25.4	26	26	60	7.98
16	34	30	42	44	85	15.94
32	50.3	48	61.6	62.3	140	31.8
48	59.9	64.4	69	72	157	47.6
64	74	76	80	82	236	63.3

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 620

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.8	15.33	16	23.7	25	1
64	99	97	111	200	320	63.3
128	146	130	170	324	560	125.6
256	243	220	280	620	1260	246.6
384	360	330	405	1100	2200	363
512	500	440	570	1500	3900	476

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 670

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	15.3	16	24	25	1
64	95	96	106	180	310	63.3
128	133	120	150	290	530	125.7
256	230	220	256	550	1200	247
384	337	324	373	860	2040	364.5
512	480	430	500	1380	3200	478

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 700

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	15.4	17	24	25	1
64	93	93	100	170	295	63.4
128	130	120	160	270	500	125.8
256	220	200	250	540	1170	247
384	327	314.4	360	790	2020	365
512	453	423	480	1290	3050	479

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 700

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	16	15.3	16	24	25	1
64	94	94	105	165	300	63.4
128	132	120	160	270	500	125.7
256	224	213	253	500	1130	247.4
384	320	310	360	730	1960	366
512	440	420	480	1200	3000	479

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	1200
True	32	100

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	1400
True	32	100

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	1500
True	32	100

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	1500
True	32	100

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 102

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.3	12	12.4	12.7	23.5	1
8	22.6	22.2	23	23.4	49	7.98
16	30.5	29.6	30	34	70	15.95
32	53	50	60.3	62	153	31.8
48	79	84.4	90.4	92	250	47.55

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 104

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.5	12.3	12.7	13	24	1
8	22.9	22.3	23.2	24	50	7.98
16	32	30.5	36	39	80	15.94
32	54	58	62	63.6	150	31.8
48	83	89	92	94	240	47.6

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 103

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.6	12.35	12.78	13.2	24	1
8	23	22.4	23.4	25	50	7.98
16	33	31	38	40	80	15.94
32	56	60	65	70	150	31.8
48	84	89	90	100	250	47.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 450

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.6	14.2	15	21.5	23	1
64	110	110	120	190	300	63.4
128	190	194	216	340	560	125.6
256	344	340	397	700	1420	246.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 460

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.6	14.2	15	22	23	1
64	107	110	120	170	278	63.4
128	194	196	215	330	540	125.6
256	340	340	390	640	1380	247

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 470

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.6	14.2	15	21.7	23	1
64	110	114	125	180	300	63.4
128	191	190	220	340	550	125.7
256	340	345	396	630	1360	246.7

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	1160
True	32	120

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	1190
True	32	110

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	1190
True	32	105

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 107

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	18.7	18.2	18.66	19	39.4	0.999
8	37	34.5	40	50	86	7.97
16	50	50	62.4	64	130	15.9
32	60	65	67.5	68.4	160	31.75
48	86	89.2	93.2	95	280	47.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 109

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19	18.5	18.9	20	40	0.999
8	37	35.2	40	50	90	7.97
16	49	44	62.3	64	130	15.9
32	60	65.2	68	68.7	160	31.74
48	86	89.4	93	97	270	47.45

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 108

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19.7	19	20	20.4	45	0.998
8	40.6	38.3	50	52	100	7.97
16	50	50	64	65	130	15.9
32	61	65	70	70	160	31.74
48	88	90.7	95	98	275	47.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 440

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20.6	20	20	31	32	0.999
64	140	120	170	250	420	63.1
128	204	200	217	400	734	124.8
256	364	340	400	800	2080	243.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 447

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20.7	20.1	21	31	32	0.999
64	130	120	170	200	400	63.1
128	200	200	220	390	730	124.9
256	360	340	390	760	2040	244

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 450

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	22	21.4	22.3	35	37	0.999
64	150	150	180	240	410	63.1
128	200	200	230	390	730	124.9
256	360	350	396	750	2040	244

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	980
True	32	110

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	1000
True	32	110

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	1000
True	32	100

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 87

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.5	13.26	13.6	14	26	1
8	24.1	23.5	24.3	25	53	7.98
16	32.6	31.4	35	40	80	15.94
32	59	57	68	70	180	31.75
48	90	98	105	110	300	47.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 87

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.8	13.5	13.9	14.2	27	1
8	25	24	30	30	54	7.98
16	36	33	42	42.5	80	15.94
32	60	66	72	74	180	31.76
48	97	102	106	110	290	47.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 87

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.6	14.26	14.9	15.2	32	0.999
8	28	28	30	33	66	7.98
16	38	40	44	45	100	15.93
32	63	67	75	80	200	31.75
48	97	102	107	110	295	47.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 370

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.6	15.2	16	23	24	1
64	128	130	142	210	340	63.3
128	230	230	257	400	666	125.2
256	428	410	493	840	2200	245

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 380

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.6	15.2	16	22.7	24	1
64	128	130	138	200	330	63.3
128	230	234	260	400	660	125.2
256	430	420	480	780	2150	245

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 375

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	17	16.4	17.5	27	30	1
64	140	140	160	200	340	63.3
128	230	230	270	400	660	125.2
256	426	424	490	780	2150	245

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	900
True	32	110

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	930
True	32	110

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	920
True	32	100

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 210

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.5	12.3	13	13.3	24	1
8	22	21	20	25	47	7.98
16	24.7	24.1	24.9	25.3	65	15.95
32	40	39	47.5	48.6	109	31.83
48	51	50	59	61	140	47.7

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 218

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.4	12.3	13	13	22.3	1
8	22.4	21.3	27	30	45	7.99
16	27	25	34	35	60	15.95
32	38	36.8	46	47	100	31.84
48	48	44.7	57.7	59	130	47.7

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 220

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	12.7	12.6	13	13.2	22.4	1
8	23	21.8	27	30	45.6	7.99
16	27	25	34	35	60	15.96
32	39.4	38	47.2	48	100	31.85
48	50	50	59	60.2	132	47.7

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 216

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.9	13.4	13.6	23	1
8	22.4	22	23.1	24	47	7.98
16	26	24.8	30	35	60	15.96
32	41	40	48.4	49.4	100	31.84
48	53	56	61	63	128	47.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 790

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	13	13.4	19	20	1
64	74	73	80	140	230	63.5
128	117	109	127	243	430	126.1
256	197	190	230	480	930	248.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 834

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14	13.6	14	21	22	1
64	71.3	71.3	77	122	217	63.5
128	110	103	118	220	408	126.2
256	188.6	180	216	430	800	249

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 860

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.4	14	15	21.2	23	1
64	73	71	76	120	220	63.5
128	112	107	117	211	400	126.2
256	186	180	213	410	790	249

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 860

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14.5	15	21.5	22.4	1
64	71	71	77	117	210	63.5
128	110	104	120	210	400	126.3
256	184	170	210	410	790	249

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	1700
True	32	120

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	1800
True	32	120

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	1850
True	32	110

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	1900
True	32	130

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 92

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19	19	19.45	19.6	39	0.999
8	39.6	38.6	39.3	40	90	7.97
16	57	50	66	70	140	15.9
32	82	90	94	96	270	31.7
48	114	120	127	131.5	350	47.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 95

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19	19	19.5	19.6	39	0.999
8	40	39	40	41	90	7.97
16	49	46	60	62	120	15.92
32	78	80	92.2	93.3	256	31.7
48	113	121	126	130	320	47.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 97

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	19.5	19	19.7	19.8	40	0.999
8	40	39	40	41	90	7.97
16	50	47	60	65	120	15.92
32	78	87	91.5	92.5	240	31.7
48	111	120	124	130	300	47.4

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 99

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	19	19.7	20	41	0.999
8	40	40	40	50	90	7.97
16	50	50	60	65	110	15.92
32	75	75	91	92	240	31.7
48	107	116	121	125	298	47.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 350

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20	20	20.6	29	30	1
64	160	158	180	300	450	63
128	260	250	300	560	900	124.5
256	474	430	535	1200	2400	243

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 380

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20.4	20	21	29	30.6	1
64	155	153.5	170	270	420	63.1
128	240	240	280	470	760	124.8
256	435	410	500	1100	2200	243.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 400

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20.6	20	21.3	29.5	31	1
64	148	148	161	240	390	63.2
128	240	240	270	440	730	124.9
256	420	400	480	880	2140	244

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 410

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	20.6	20	20.9	30	31	1
64	144	146	156	224	370	63.2
128	240	240	260	410	700	125
256	406	396	460	777	2100	244.5

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	760
True	32	60

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	840
True	32	60

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	900
True	32	56

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	930
True	32	50

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 39

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	33.7	32.7	33.1	33.3	77.8	0.998
8	55	55	60.8	63	146	7.95
16	63	59.4	70	70	180	15.87
32	140	125	160	200	470	31.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 39

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	35.1	34	34.7	34.86	79	0.998
8	56	56	62.6	63.7	150	7.95
16	61.5	58.7	63	64	160	15.88
32	145	129	150	200	470	31.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 39

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	34.1	33.2	33.5	33.6	79	0.998
8	56	56	61.6	63	140	7.95
16	62	59	63	65	159	15.88
32	145	130	160	210	450	31.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 39

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	34.6	33.65	34.05	34.2	79	0.998
8	57	57	63	65	150	7.95
16	61.5	59	62.4	63.5	160	15.88
32	142	130	160	200	460	31.5

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 166

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	29	28	28.4	47.4	48.3	0.999
64	250	266	290	440	716	62.5
128	550	510	600	1100	2600	121.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 170

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	29.5	28.3	29	48	49	0.998
64	244	260	284	430	700	62.5
128	540	516	580	970	2370	122

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 173

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	29.5	28.4	29	48.4	49	0.998
64	240	250	280	420	696	62.5
128	530	510	580	900	2360	122.2

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 174

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	30.8	29.6	30.3	50.3	51	0.998
64	240	250	280	420	700	62.5
128	530	500	580	1000	2360	122.2

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	410
True	32	80

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	423
True	32	80

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	430
True	32	73

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	430
True	32	80

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 87

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.1	12.9	13.25	13.4	26	1
8	22.4	21.7	22.6	25	52	7.98
16	30	29	30	30	80	15.94
32	54	50.7	61	62.4	180	31.74
48	82	85	96	98	290	47.45

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 87

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.3	13	13.4	13.7	26.2	1
8	22.6	21.7	24	25	51	7.98
16	30	30	35	40	76	15.94
32	54	52	62	63	170	31.74
48	86	92	97.6	100	285	47.5

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 88

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.45	13.2	13.6	13.8	26.5	1
8	26	25	30	30	57	7.98
16	35	36	41.25	42	90	15.94
32	55.5	58	63.4	64.6	180	31.74
48	88	93	100	104	290	47.4

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 370

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.4	15	16	23	24	1
64	110	110	126	204	360	63.2
128	220	220	247	410	720	125
256	430	410	480	900	2240	244

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 374

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15	14.8	17	23	24	1
64	115	118	127	197	350	63.2
128	220	220	245	400	710	125
256	430	410	480	790	2220	244

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 374

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	15.5	14.9	20	23	26	1
64	130	130	140	210	370	63.2
128	220	220	250	400	700	125
256	424	410	486	800	2230	244

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	900
True	32	100

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	930
True	32	104

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	930
True	32	95

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 210

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.1	10.9	11.33	11.5	20	1
8	18	17.4	19	20	39	7.99
16	23	22	27	28	55	15.96
32	36	38	41	42	100	31.86
48	48	50	53	55	130	47.7

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 210

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.1	11	11.4	11.7	20	1
8	18	18	20	20	38	7.99
16	24	24	29	30	56	15.96
32	35	37.3	41	42	94	31.86
48	47	50	53	55	120	47.7

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 210

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.1	11	11.3	11.5	20	1
8	18.6	18	20	20	39	7.99
16	24	23	28.6	29.4	50	15.96
32	37	39	42	42.8	100	31.86
48	49	51	54	56	120	47.7

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 212

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	11.7	11.4	12.2	12.8	23.4	1
8	21.4	21	23	24	50	7.98
16	26	26	30	32	60	15.96
32	36	38	42	43	93	31.86
48	49	51.4	55	57	120	47.7

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 820

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13.3	13	16	19.4	20	1
64	67	64	74	130	200	63.6
128	110	104	121	230	380	126.3
256	190	183	222.5	450	750	249.6

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 850

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.7	13.3	19.5	20	1
64	65.5	63.2	73	120	190	63.6
128	105	100	114	200	360	126.4
256	180	170	210	400	702	250

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 860

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	13	12.7	13.7	20	20	1
64	66	64.6	74	116	190	63.6
128	106	100	120	200	360	126.4
256	180	170	210	380	700	250

Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 800

# of streams	Latency (ms)					Throughput (RTFX)
	avg	p50	p90	p95	p99
1	14.2	13.7	14.5	23	25	1
64	77	73	90	130	200	63.5
128	110	105	130	200	360	126.4
256	180	180	220	370	680	250

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	1950
True	32	140

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	2200
True	32	130

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	2100
True	32	130

Language model: n-gram

Speaker Diarization	# of streams	Throughput (RTFX)
False	32	2100
True	32	110

On-Prem Hardware Specifications#

GPU
NVIDIA DGX A100 40 GB
CPU
Model	AMD EPYC 7742 64-Core Processor
Thread(s) per core	2
Socket(s)	2
Core(s) per socket	64
NUMA node(s)	8
Frequency boost	enabled
CPU max MHz	2250
CPU min MHz	1500
RAM
Model	Micron DDR4 36ASF8G72PZ-3G2B2 3200MHz
Configured Memory Speed	2933 MT/s
RAM Size	32x64GB (2048GB Total)

GPU
NVIDIA A30
CPU
Model	AMD EPYC 7742 64-Core Processor
Thread(s) per core	1
Socket(s)	2
Core(s) per socket	64
NUMA node(s)	2
Frequency boost	disabled
CPU max MHz	2250.0000
CPU min MHz	1500.0000
RAM
Model	Samsung DDR4 M393A4K40DB3-CWE 3200MHz
Configured Memory Speed	3200 MT/s
RAM Size	32x64GB (2048GB Total)

GPU
NVIDIA A10
CPU
Model	AMD EPYC 7763 64-Core Processor
Thread(s) per core	1
Socket(s)	2
Core(s) per socket	64
NUMA node(s)	8
Frequency boost	enabled
CPU max MHz	2450
CPU min MHz	1500
RAM
Model	Samsung DDR4 M393A4K40DB3-CWE 3200 MHz
Configured Memory Speed	3200 MT/s
RAM Size	16x32GB (512GB Total)

GPU
NVIDIA V100 SXM2 16 GB
CPU
Model	Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
Thread(s) per core	2
Socket(s)	2
Core(s) per socket	20
NUMA node(s)	2
CPU max MHz	3600
CPU min MHz	1200
RAM
Model	Micron DDR4 36ASF4G72PZ-2G6D1 2667MHz
Configured Memory Speed	2133 MT/s
RAM Size	16x32GB (512GB Total)

GPU
NVIDIA T4
CPU
Model	Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz
Thread(s) per core	2
Socket(s)	2
Core(s) per socket	18
NUMA node(s)	2
CPU max MHz	3900
CPU min MHz	1000
RAM
Model	Samsung DDR4 M393A2K43BB1-CTD 2666MHz
Configured Memory Speed	2666 MT/s
RAM Size	24x16GB (384GB Total)

NVIDIA Riva

Performance

Contents

Performance#

Evaluation Process#

Results#

On-Prem Hardware Specifications#