Performance
Contents
Performance#
Evaluation Process#
This section shows the latency and throughput numbers for streaming and offline configurations of the Riva ASR service on different GPUs. These numbers were captured after the preconfigured ASR pipelines from our Quick Start scripts were deployed. The Jasper, QuartzNet, Conformer, and Citrinet-1024 acoustic models were tested.
In streaming mode, the client and the server used audio chunks of the same duration (100ms, 160ms, and 800ms depending on the server configuration). Refer to the Results section for the chunk size value to use.
The Riva streaming client riva_streaming_asr_client
,
provided in the Riva image, was used with the --simulate_realtime
flag to
simulate transcription from a microphone, where each stream was doing three iterations
over a sample audio file (1272-135031-0000.wav
) from the LibriSpeech dev-clean dataset.
The Librispeech datasets can be obtained from https://www.openslr.org/12.
The source code for the riva_streaming_asr_client
can be obtained from https://github.com/nvidia-riva/cpp-clients.
The command used to measure performance was:
riva_streaming_asr_client \
--chunk_duration_ms=<chunk_duration> \
--simulate_realtime=true \
--automatic_punctuation=true \
--num_parallel_requests=<num_streams> \
--word_time_offsets=true \
--print_transcripts=false \
--interim_results=false \
--num_iterations=<3*num_streams> \
--audio_file=1272-135031-0000.wav \
--output_filename=/tmp/output.json
The riva_streaming_asr_client
returns the following latency measurements:
intermediate latency
: latency of responses returned withis_final == false
final latency
: latency of responses returned withis_final == true
latency
: the overall latency of all returned responses. This is what is tabulated in the following tables.
Refer to the following diagram for a schematic representation of the different latencies measured by the Riva streaming ASR client.
In offline mode, the command used to measure maximum throughput was:
riva_asr_client \
--automatic_punctuation=true \
--num_parallel_requests=32 \
--word_time_offsets=true \
--print_transcripts=false \
--num_iterations=96 \
--audio_file=1272-135031-0000x5.wav \
--output_filename=/tmp/output.json
where 5x_1272-135031-0000.wav
is simply the 1272-135031-0000.wav
audio file concatenated five times.
The source code for the riva_asr_client
can be obtained from: https://github.com/nvidia-riva/cpp-clients
Results#
Latencies and throughput measurements for streaming and offline configurations are reported in the following tables. Throughput is measured in RTFX (duration of audio transcribed / computation time).
Note
If the language model is none
, the inference is performed with a greedy decoder. If the language model is n-gram
,
then a beam decoder was used.
For specifications of the hardware on which these measurements were collected, refer to the Hardware Specifications section.
160
n-gram
247
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
9.91 |
9.685 |
9.96 |
10.15 |
16.74 |
0.999460 |
8 |
14.48 |
13.9 |
14.95 |
15.5 |
33.8 |
7.9864 |
16 |
23.0 |
22.97 |
25.1 |
25.81 |
59.5 |
15.9493 |
32 |
34.9 |
33.9 |
36.3 |
37.4 |
100 |
31.814 |
48 |
46.2 |
44.5 |
48.8 |
50.0 |
142 |
47.623 |
64 |
55.4 |
53.0 |
58.1 |
60.8 |
202 |
63.35 |
800
n-gram
979
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
10.8 |
10.24 |
10.8 |
16.1 |
16.05 |
0.999480 |
64 |
63.9 |
55.9 |
61.3 |
135 |
263 |
63.42 |
128 |
107.5 |
94.2 |
108 |
237 |
474 |
125.92 |
256 |
172.3 |
147.8 |
174 |
406 |
964 |
248.34 |
384 |
242 |
207.0 |
241 |
585.0 |
1428 |
367.27 |
512 |
318 |
272 |
320 |
785 |
2104 |
482.2 |
768 |
484.0 |
400 |
477 |
1419 |
3880 |
702.7 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
4000 |
160
n-gram
262
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
9.533 |
9.527 |
9.81 |
9.889 |
11.7 |
0.999663 |
8 |
13.71 |
13.58 |
14.36 |
15.00 |
18.9 |
7.99607 |
16 |
19.11 |
18.59 |
24.2 |
24.7 |
24.26 |
15.9870 |
32 |
34.47 |
34.6 |
36.2 |
37.15 |
40.7 |
31.9590 |
48 |
42.95 |
42.9 |
47.0 |
48.0 |
51.5 |
47.920 |
64 |
53.9 |
53.6 |
59.1 |
60.7 |
67.7 |
63.867 |
800
n-gram
1235
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
10.3 |
10.28 |
11.2 |
11.46 |
12.0 |
1.000 |
64 |
55.4 |
55.3 |
62.1 |
63 |
66.8 |
63.871 |
128 |
95.6 |
96.0 |
108.3 |
112.0 |
118 |
127.52 |
256 |
153 |
157 |
188 |
195 |
198.9 |
254.43 |
384 |
189 |
202.9 |
236 |
244 |
258 |
380.80 |
512 |
238.9 |
265.2 |
306 |
316 |
335 |
506.43 |
768 |
351 |
397.3 |
462 |
483 |
524 |
754.7 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1692.2 |
160
n-gram
228
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
10.06 |
9.94 |
10.27 |
10.8 |
13.0 |
1.000 |
8 |
16.20 |
15.99 |
16.8 |
17.08 |
23.1 |
7.9942 |
16 |
20.92 |
20.16 |
28.1 |
29.5 |
32.4 |
15.982 |
32 |
42.54 |
42.4 |
44.5 |
46.3 |
50.1 |
31.9443 |
48 |
52.6 |
53.3 |
61.2 |
63.1 |
66.1 |
47.888 |
64 |
59.6 |
59.9 |
66.4 |
68.7 |
74.2 |
63.831 |
800
n-gram
990
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
10.8 |
10.67 |
11.4 |
11.7 |
11.9 |
1.000 |
64 |
68.6 |
70.8 |
79.4 |
81.0 |
85.6 |
63.804 |
128 |
104 |
104 |
136 |
136.0 |
153.3 |
127.21 |
256 |
176 |
184.2 |
250.50 |
286 |
315 |
253.11 |
384 |
251 |
262 |
356 |
461 |
540 |
376.7 |
512 |
312 |
334.0 |
483 |
610 |
645 |
499.2 |
768 |
493 |
501 |
743 |
1060 |
1400 |
737.1 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1207 |
160
n-gram
188
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
11.18 |
11.074 |
11.31 |
11.44 |
14.3 |
0.999540 |
8 |
20.2 |
20.2 |
20.76 |
21.01 |
28.9 |
7.99257 |
16 |
31.2 |
24.89 |
37.4 |
37.8 |
45.4 |
15.9760 |
32 |
52.0 |
53.0 |
55.1 |
55.7 |
59.5 |
31.9307 |
48 |
63.7 |
63.7 |
70.0 |
71.68 |
75.0 |
47.875 |
64 |
71.2 |
71.9 |
77.8 |
79.4 |
85.0 |
63.801 |
800
n-gram
827
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.4 |
12.216 |
12.80 |
13.50 |
14.48 |
0.999523 |
64 |
88.3 |
89.1 |
97.3 |
101 |
106 |
63.786 |
128 |
136 |
133 |
172 |
190.0 |
211.3 |
127.11 |
256 |
209 |
216 |
268 |
423 |
457 |
252.09 |
384 |
291 |
314 |
378 |
625 |
674 |
375.55 |
512 |
382.1 |
410.6 |
496 |
870 |
963 |
496.5 |
768 |
665 |
626 |
1347 |
1614 |
2053 |
732.9 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1192 |
160
n-gram
147
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.23 |
11.75 |
17.3 |
28.8 |
35.60 |
0.999577 |
8 |
24.12 |
21.5 |
28.6 |
55.6 |
71.2 |
7.9924 |
16 |
37.9 |
35.4 |
44.8 |
70.0 |
94.3 |
15.9747 |
32 |
66.7 |
64.6 |
73.6 |
93 |
113.8 |
31.9220 |
48 |
87.5 |
84.8 |
100.76 |
116 |
153 |
47.845 |
64 |
98.8 |
97.5 |
113 |
131 |
176 |
63.7573 |
800
n-gram
588
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.5 |
17.74 |
33.8 |
34.9 |
34.91 |
0.999567 |
64 |
116 |
110 |
163 |
170 |
182.5 |
63.758 |
128 |
188 |
178 |
252 |
274 |
283 |
127.07 |
256 |
302 |
308 |
399 |
424 |
488 |
252.5 |
384 |
463 |
472 |
689 |
747 |
865 |
373.6 |
512 |
598 |
610 |
855 |
1070 |
1200 |
492.5 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
616 |
160
n-gram
183
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
11.14 |
11.05 |
11.46 |
11.85 |
12.54 |
0.999510 |
8 |
19.19 |
18.990 |
20.07 |
20.42 |
24.3 |
7.9926 |
16 |
29.2 |
24.37 |
37.9 |
38.3 |
41.3 |
15.9743 |
32 |
49 |
50 |
54.8 |
56.9 |
65.5 |
31.913 |
48 |
62.7 |
61.64 |
70.1 |
73.5 |
86 |
47.851 |
64 |
70.1 |
70.9 |
78.1 |
80.7 |
104 |
63.745 |
800
n-gram
845
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.0 |
11.89 |
12.5 |
12.7 |
12.7 |
0.999503 |
64 |
86.3 |
88.0 |
97.5 |
99.7 |
105 |
63.76 |
128 |
114.6 |
119 |
146 |
152 |
161 |
127.30 |
256 |
183.5 |
206 |
236 |
242 |
259 |
253.44 |
384 |
255.0 |
302 |
339.2 |
344.1 |
356.4 |
378.69 |
512 |
343.0 |
399.6 |
464 |
511 |
553 |
501.75 |
768 |
574 |
620 |
821 |
937 |
1047 |
740 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1085.2 |
160
179
192
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
13.92 |
13.571 |
13.94 |
14.18 |
28.4 |
0.99909 |
n-gram |
8 |
26.19 |
25.50 |
26.11 |
26.96 |
62 |
7.9792 |
n-gram |
16 |
37.1 |
33.0 |
44.82 |
45.42 |
89.4 |
15.93 |
n-gram |
32 |
52.4 |
51.3 |
63.0 |
64.0 |
138 |
31.783 |
n-gram |
48 |
61.9 |
64.6 |
69.1 |
72.2 |
150 |
47.611 |
n-gram |
64 |
75.5 |
76.3 |
80.7 |
82.9 |
233 |
63.314 |
none |
1 |
13.11 |
12.821 |
12.96 |
13.18 |
22.0 |
0.999293 |
none |
8 |
25.2 |
24.54 |
24.92 |
25.3 |
53 |
7.9816 |
none |
16 |
33.8 |
29.8 |
43.7 |
44.4 |
77 |
15.9383 |
none |
32 |
49.2 |
50.1 |
60.85 |
62.7 |
121 |
31.799 |
none |
48 |
56.2 |
62.0 |
66.4 |
70.3 |
141 |
47.622 |
none |
64 |
70.3 |
73.2 |
75.6 |
77.2 |
187 |
63.343 |
800
687
751
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
16.0 |
15.38 |
16.3 |
23.6 |
24.86 |
0.999253 |
n-gram |
64 |
97.2 |
95.7 |
106.7 |
167 |
300 |
63.347 |
n-gram |
128 |
138 |
128 |
168.1 |
270 |
525 |
125.69 |
n-gram |
256 |
233 |
221 |
270 |
525 |
1174 |
247.06 |
n-gram |
384 |
334 |
327 |
379 |
743 |
1990 |
364.68 |
n-gram |
512 |
469 |
436 |
504 |
1243 |
3057 |
478.37 |
none |
1 |
13.04 |
12.672 |
13.14 |
17.084 |
17.7 |
0.999440 |
none |
64 |
86.0 |
88.2 |
93.2 |
158.6 |
288 |
63.373 |
none |
128 |
112 |
93 |
129 |
249 |
491 |
125.853 |
none |
256 |
203 |
194 |
239 |
470 |
1086 |
247.67 |
none |
384 |
303 |
304.4 |
344 |
691 |
1710 |
365.59 |
none |
512 |
422 |
411 |
461 |
1042 |
2868 |
480.4 |
Language model |
# of streams |
Speaker Diarization |
Throughput (RTFX) |
---|---|---|---|
n-gram |
32 |
False |
1460 |
n-gram |
32 |
True |
62.1 |
none |
32 |
False |
1500 |
none |
32 |
True |
60.4 |
160
n-gram
182
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.48 |
13.33 |
13.802 |
14.07 |
20.0 |
0.99950 |
8 |
25.68 |
25.01 |
25.71 |
26.4 |
52.0 |
7.99310 |
16 |
34.5 |
30.8 |
44.5 |
45.1 |
77.4 |
15.98 |
32 |
50.0 |
50.3 |
62.1 |
63.0 |
127.5 |
31.945 |
48 |
61.62 |
64.8 |
69.9 |
74.0 |
182.8 |
47.8857 |
64 |
74.6 |
75.3 |
81.16 |
87.67 |
272 |
63.825 |
800
n-gram
704
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.6 |
14.17 |
14.97 |
20.55 |
20.697 |
0.999533 |
64 |
93.3 |
93.5 |
108 |
175 |
296 |
63.8177 |
128 |
141 |
128 |
145.5 |
292 |
516 |
127.357 |
256 |
228.3 |
211.5 |
271 |
585.1 |
1192 |
253.45 |
384 |
386 |
318 |
785 |
1311 |
2102 |
371.31 |
512 |
615 |
429 |
1718 |
2440 |
3390 |
482.9 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
970 |
160
n-gram
162
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.88 |
13.85 |
14.31 |
14.42 |
16.67 |
0.999467 |
8 |
27.5 |
27.34 |
28.32 |
28.4 |
38.3 |
7.99170 |
16 |
35.0 |
35.4 |
36.0 |
48.5 |
51.4 |
15.976 |
32 |
57.1 |
64.1 |
69.96 |
70.98 |
74 |
31.910 |
48 |
69.1 |
76.1 |
81.4 |
83.1 |
87.5 |
47.854 |
64 |
86.62 |
93.3 |
99.4 |
100.8 |
108 |
63.753 |
800
n-gram
756
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.7 |
14.79 |
16.2 |
16.9 |
16.734 |
0.999470 |
64 |
92 |
101 |
111.6 |
115.4 |
137 |
63.740 |
128 |
139 |
135.4 |
181 |
201 |
273 |
126.99 |
256 |
231 |
234 |
316 |
413 |
431 |
251.81 |
384 |
338 |
340.8 |
646 |
707 |
739 |
373.96 |
512 |
458 |
468 |
810 |
971 |
1054 |
494.0 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1940 |
160
n-gram
188
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.48 |
13.370 |
13.66 |
13.84 |
16.23 |
0.999 |
8 |
25.19 |
25.13 |
25.59 |
25.96 |
31.4 |
7.9904 |
16 |
32.9 |
30.00 |
43.5 |
44.55 |
45.8 |
15.974 |
32 |
50.34 |
57.2 |
61.9 |
62.7 |
68.9 |
31.9223 |
48 |
58.7 |
64.93 |
68.2 |
69.7 |
81.4 |
47.8613 |
64 |
70.5 |
75.6 |
79.7 |
80.7 |
85.1 |
63.778 |
800
n-gram
804
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.6 |
14.34 |
14.64 |
15.8 |
16.4 |
0.999463 |
64 |
85.4 |
92.7 |
102.5 |
108.0 |
118 |
63.767 |
128 |
120 |
119 |
141.3 |
186 |
223 |
127.16 |
256 |
196.8 |
204 |
257 |
420 |
469 |
252.26 |
384 |
285 |
310 |
412.76 |
650 |
670 |
375.5 |
512 |
392 |
419 |
547 |
859 |
999 |
496.4 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
2850 |
160
none
93
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
15.09 |
14.87 |
15.36 |
15.6 |
20.2 |
0.998660 |
8 |
32.3 |
31.85 |
33.1 |
41.0 |
53.6 |
7.966 |
16 |
48.3 |
48.1 |
58 |
61 |
84.3 |
15.889 |
32 |
72.5 |
77.1 |
88.6 |
91.5 |
146 |
31.612 |
48 |
91.6 |
98.2 |
105.8 |
112 |
184.6 |
47.250 |
64 |
109 |
115 |
125 |
136 |
269 |
62.76 |
800
none
382
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
18.26 |
17.57 |
20.16 |
20.4 |
20.40 |
0.998633 |
64 |
126 |
123.5 |
150.3 |
242 |
306 |
62.66 |
128 |
214 |
205 |
298 |
450 |
575 |
123.02 |
256 |
408 |
369 |
560 |
909 |
1320 |
237.3 |
none
# of streams |
Throughput (RTFX) |
---|---|
32 |
867 |
160
n-gram
190
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.58 |
13.482 |
13.99 |
14.50 |
17.1 |
0.999373 |
8 |
25.55 |
25.42 |
26.18 |
26.69 |
32.6 |
7.9908 |
16 |
34.8 |
30.93 |
44.4 |
45.03 |
48.0 |
15.970 |
32 |
50.3 |
56.1 |
62.5 |
63.5 |
71 |
31.900 |
48 |
58.0 |
63.8 |
68.6 |
70.4 |
97 |
47.813 |
64 |
69.7 |
74.1 |
79.4 |
81.8 |
117.7 |
63.705 |
800
n-gram
856
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.6 |
14.54 |
15.5 |
16.1 |
16.6 |
0.999387 |
64 |
84.9 |
92.4 |
99.1 |
101.6 |
109 |
63.742 |
128 |
110 |
102.8 |
132.1 |
137.0 |
159 |
127.23 |
256 |
172.1 |
184 |
236.9 |
243.2 |
255.3 |
253.41 |
384 |
246 |
295.2 |
341.5 |
348.5 |
377 |
378.55 |
512 |
340 |
408 |
459.5 |
494.5 |
543 |
501.61 |
768 |
538 |
607.5 |
691 |
890 |
1150 |
740.8 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
2050 |
160
n-gram
175
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.46 |
13.13 |
13.67 |
16.0 |
20.1 |
0.999343 |
8 |
25.94 |
24.75 |
25.85 |
32.1 |
57.2 |
7.9817 |
16 |
34.3 |
29.5 |
44.0 |
45.5 |
94.54 |
15.932 |
32 |
50.6 |
49.8 |
61.98 |
76 |
160.3 |
31.782 |
48 |
62.5 |
63.4 |
69.6 |
81.8 |
242.6 |
47.586 |
64 |
77.7 |
75.1 |
82.7 |
127.9 |
301.7 |
63.316 |
800
n-gram
509
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
15.3 |
13.99 |
20.2 |
20.3 |
20.23 |
0.999323 |
64 |
107.6 |
86 |
231 |
284 |
323.4 |
63.300 |
128 |
167 |
123 |
384 |
486 |
566 |
125.64 |
256 |
309 |
223.4 |
717 |
1085 |
1251 |
246.91 |
384 |
526 |
341 |
1355 |
1640 |
2150 |
364.4 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
984 |
160
n-gram
151
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.76 |
13.70 |
14.32 |
14.64 |
15.66 |
0.999003 |
8 |
26.6 |
26.28 |
27.72 |
29.3 |
37.6 |
7.98427 |
16 |
37.3 |
35.9 |
46.0 |
46.9 |
49.5 |
15.9453 |
32 |
56.8 |
63.4 |
69.0 |
69.9 |
73.1 |
31.842 |
48 |
67.3 |
73.1 |
78.7 |
81 |
80.7 |
47.7320 |
64 |
81.4 |
87.4 |
94.2 |
95.5 |
102.4 |
63.557 |
800
n-gram
739
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.8 |
14.53 |
15.62 |
15.58 |
15.58 |
0.998833 |
64 |
94.7 |
102.7 |
111.1 |
114.0 |
117.1 |
63.448 |
128 |
131.9 |
132.1 |
173.2 |
176.6 |
199 |
126.22 |
256 |
218 |
235 |
273 |
271.9 |
283 |
250.49 |
384 |
303 |
341 |
396 |
413 |
437 |
372.07 |
512 |
404 |
465 |
535 |
559 |
633 |
491.7 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1590 |
100
n-gram
286
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
7.14 |
6.795 |
7.95 |
8.8 |
14.82 |
0.999453 |
8 |
10.08 |
9.25 |
10.95 |
12.22 |
36.8 |
7.9878 |
16 |
14.2 |
12.00 |
16.4 |
18.53 |
60.3 |
15.9563 |
32 |
16.57 |
14.67 |
20.2 |
23.0 |
102 |
31.854 |
48 |
22.7 |
19.3 |
25.9 |
29.8 |
153.8 |
47.689 |
64 |
27.6 |
22.8 |
30.0 |
37.0 |
196.2 |
63.470 |
800
n-gram
1013
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.16 |
11.753 |
16.54 |
16.99 |
18.0 |
0.999470 |
64 |
55.1 |
40.2 |
111.2 |
197.1 |
262.6 |
63.444 |
128 |
85.1 |
56.4 |
192 |
348 |
470 |
126.02 |
256 |
145.5 |
85.5 |
351.8 |
656 |
942 |
248.57 |
384 |
206.6 |
112 |
551 |
993 |
1358 |
368.3 |
512 |
282 |
146 |
840 |
1322 |
1893 |
483.98 |
768 |
495 |
198 |
1620 |
2333 |
3170 |
699.6 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
2110 |
100
n-gram
175
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.14 |
12.85 |
13.79 |
14.9 |
21.1 |
0.99929 |
8 |
15.22 |
14.56 |
16.4 |
18.112 |
36.6 |
7.9870 |
16 |
18.74 |
17.07 |
25.4 |
26.1 |
58.6 |
15.9537 |
32 |
34.3 |
33.5 |
39.3 |
41.5 |
97.9 |
31.8387 |
48 |
42.8 |
41.3 |
47.7 |
52.3 |
165 |
47.668 |
64 |
50.0 |
46.1 |
55.6 |
59.8 |
235 |
63.46 |
800
n-gram
825
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
17.5 |
16.7 |
21.6 |
24.8 |
25.5 |
0.999343 |
64 |
71.6 |
56.9 |
129.6 |
212 |
276 |
63.418 |
128 |
123.0 |
96.11 |
232 |
385 |
503 |
125.94 |
256 |
199 |
142 |
395 |
698 |
1037 |
248.25 |
384 |
279.0 |
186 |
644 |
1138 |
1512 |
367.4 |
512 |
388 |
240 |
1114 |
1508 |
2184 |
482.9 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
2030 |
160
n-gram
138
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.839 |
13.537 |
13.86 |
13.96 |
26.31 |
0.999143 |
8 |
23.4 |
22.7 |
23.4 |
24.3 |
56.5 |
7.9772 |
16 |
40.48 |
40.9 |
42.71 |
43.51 |
102.4 |
15.915 |
32 |
60.2 |
58.9 |
65.2 |
67.3 |
175 |
31.706 |
48 |
64.3 |
61.3 |
68.1 |
70.0 |
210 |
47.487 |
64 |
83 |
76 |
88.1 |
91.2 |
284 |
63.11 |
800
n-gram
576
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.88 |
14.2 |
15.0 |
22.7 |
23.6 |
0.99924 |
64 |
106.1 |
96.4 |
110.3 |
207 |
381 |
63.15 |
128 |
149 |
135 |
152 |
305 |
653 |
125.14 |
256 |
274.5 |
247.8 |
281.5 |
608 |
1474 |
244.49 |
384 |
422 |
373 |
420 |
1059 |
2849 |
358.6 |
512 |
620 |
512 |
602 |
1549 |
4750 |
467.25 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
2755 |
160
n-gram
148
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.36 |
13.40 |
13.74 |
13.94 |
15.23 |
0.999523 |
8 |
23.7 |
23.2 |
24.12 |
29.2 |
31.4 |
7.9928 |
16 |
39.22 |
39.7 |
41.6 |
42.5 |
46.0 |
15.9770000000000003 |
32 |
59.6 |
59.9 |
65.5 |
66.7 |
69 |
31.932 |
48 |
61.32 |
61.51 |
68.0 |
69.6 |
75 |
47.885 |
64 |
80.2 |
80.7 |
86.8 |
88.6 |
96.7 |
63.822 |
800
n-gram
711
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.28 |
14.38 |
15.1 |
15.57 |
15.8 |
0.999500 |
64 |
93 |
94 |
104 |
106 |
112 |
63.763 |
128 |
119.7 |
130 |
144.4 |
151 |
152.9 |
127.31 |
256 |
220 |
246.5 |
274 |
281 |
291 |
253.46 |
384 |
321 |
361 |
422 |
430 |
471 |
378.61 |
512 |
421 |
477 |
554 |
595 |
667 |
501.13 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1386.2 |
160
n-gram
126
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.83 |
13.82 |
14.05 |
14.14 |
15.104 |
0.999443 |
8 |
24.40 |
24.6 |
25.0 |
28.6 |
34.6 |
7.9914 |
16 |
41.6 |
43.1 |
44.6 |
45.1 |
47.6 |
15.9723 |
32 |
58.1 |
58.1 |
64.9 |
67.0 |
71.7 |
31.922 |
48 |
66.8 |
69.6 |
77.8 |
79.4 |
83.0 |
47.849 |
64 |
80.7 |
87 |
98 |
100 |
103 |
63.772 |
800
n-gram
623
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.73 |
14.74 |
15.5 |
16.0 |
15.948 |
0.999417 |
64 |
91 |
90 |
123 |
132 |
135.8 |
63.70 |
128 |
149.0 |
156.6 |
207 |
218 |
272 |
126.89 |
256 |
272.4 |
283.1 |
422 |
481 |
513 |
251.15 |
384 |
385 |
404.7 |
560 |
779 |
908 |
372.6 |
512 |
547 |
540 |
1040 |
1130 |
1400 |
490.82 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1020.8 |
160
n-gram
114
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
15.16 |
15.10 |
15.394 |
15.49 |
17.1 |
0.999423 |
8 |
28.70 |
28.4 |
29.35 |
29.73 |
30.5 |
7.9899 |
16 |
50 |
52.8 |
55.8 |
56.36 |
59.2 |
15.9690 |
32 |
63.5 |
63.3 |
69.0 |
70.6 |
75 |
31.918 |
48 |
84.8 |
85.3 |
90.0 |
90.9 |
96.8 |
47.841 |
64 |
104.2 |
104.4 |
113.0 |
116.1 |
127.3 |
63.69 |
800
n-gram
555
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
16.09 |
15.99 |
16.46 |
16.89 |
16.79 |
0.999397 |
64 |
100 |
102 |
116 |
127.7 |
175 |
63.717 |
128 |
170 |
176.1 |
223.2 |
291 |
302.8 |
126.69 |
256 |
324.8 |
341 |
424 |
624 |
678 |
250.33 |
384 |
451 |
475 |
573 |
990 |
1125 |
371.36 |
512 |
659 |
637 |
1260 |
1410 |
1666 |
490.30 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1011 |
160
n-gram
101
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
17.76 |
16.62 |
21.34 |
31.9 |
38.87 |
0.999380 |
8 |
39.1 |
38.3 |
46.1 |
57.4 |
81 |
7.9873 |
16 |
56.3 |
54.6 |
66.8 |
76.6 |
94 |
15.962 |
32 |
81 |
79 |
91.6 |
107 |
131.2 |
31.904 |
48 |
96.0 |
94.0 |
114.3 |
125 |
167 |
47.823 |
64 |
122.0 |
121.4 |
146.5 |
163 |
190 |
63.676 |
800
n-gram
433
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
24.50 |
21.55 |
39.06 |
39.26 |
39.9 |
0.999390 |
64 |
132.2 |
128.0 |
172.0 |
184.3 |
188.6 |
63.65 |
128 |
231 |
228 |
302 |
323 |
332 |
126.85 |
256 |
417 |
417 |
552 |
630 |
790 |
249.7 |
384 |
639 |
622 |
980 |
1180 |
1340 |
367.3 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
710 |
160
n-gram
112
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
15.39 |
15.35 |
15.83 |
16.00 |
17.0 |
0.999327 |
8 |
30.39 |
30.4 |
31.726 |
32.1 |
43.1 |
7.9895 |
16 |
54.9 |
56.1 |
61.41 |
62.2 |
65.9 |
15.960 |
32 |
64.27 |
63.6 |
70.37 |
72.6 |
80.2 |
31.903 |
48 |
83 |
84.3 |
90.9 |
92.7 |
130 |
47.757 |
64 |
105.9 |
105.38 |
113.2 |
117.5 |
168 |
63.57 |
800
n-gram
558
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
15.94 |
16.07 |
16.49 |
16.6 |
16.47 |
0.999340 |
64 |
105 |
103.7 |
132 |
146 |
157.5 |
63.69 |
128 |
164 |
177.3 |
198.9 |
205 |
216 |
126.92 |
256 |
293 |
334 |
358 |
365 |
365 |
252.3 |
384 |
443 |
501 |
597 |
713 |
771 |
373.38 |
512 |
654 |
667 |
1090 |
1245 |
1374 |
491.3 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
931.4 |
160
101
111
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
18.934 |
18.519 |
18.832 |
18.96 |
39.4 |
0.99873 |
n-gram |
8 |
40.6 |
37.4 |
50.1 |
52.85 |
87.5 |
7.9687 |
n-gram |
16 |
52.9 |
48.7 |
65.5 |
67.1 |
131.9 |
15.9070 |
n-gram |
32 |
65.9 |
69.0 |
72.30 |
74.03 |
172 |
31.7300 |
n-gram |
48 |
91.2 |
93.6 |
100.3 |
103.6 |
283 |
47.413 |
none |
1 |
18.02 |
17.58 |
17.74 |
17.85 |
33.57 |
0.998917 |
none |
8 |
37.5 |
36.2 |
37.3 |
37.4 |
82.2 |
7.9703 |
none |
16 |
48.4 |
44.2 |
61.5 |
63.8 |
121 |
15.91 |
none |
32 |
60.53 |
66.4 |
69.4 |
70.3 |
158.9 |
31.733 |
none |
48 |
84.75 |
89.2 |
93.2 |
96.2 |
276 |
47.418 |
none |
64 |
107.6 |
108.8 |
116.5 |
121.7 |
385 |
62.985 |
800
424
455
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
21.00 |
20.38 |
21.2 |
31.33 |
32.4 |
0.998983 |
n-gram |
64 |
140 |
132 |
171 |
199.7 |
390 |
63.14 |
n-gram |
128 |
209 |
201 |
228.7 |
392 |
737 |
124.79 |
n-gram |
256 |
374.4 |
358 |
416 |
771 |
2107 |
243.5 |
none |
1 |
18.24 |
17.64 |
18.3 |
25.19 |
25.9 |
0.999170 |
none |
64 |
109.7 |
111 |
123.4 |
201.2 |
373 |
63.13 |
none |
128 |
190.5 |
188 |
210.3 |
378.3 |
724 |
124.85 |
none |
256 |
335.7 |
313 |
390 |
736 |
2059.4 |
243.80 |
none |
384 |
548 |
501 |
583 |
1487 |
4641 |
356.88 |
Language model |
# of streams |
Speaker Diarization |
Throughput (RTFX) |
---|---|---|---|
n-gram |
32 |
False |
904 |
n-gram |
32 |
True |
39.8 |
none |
32 |
False |
950 |
none |
32 |
True |
40.9 |
160
n-gram
103
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
18.02 |
17.82 |
18.29 |
18.47 |
27.1 |
0.999373 |
8 |
36.8 |
35.782 |
37.30 |
45.2 |
76 |
7.9892 |
16 |
50.4 |
46.3 |
64.0 |
65.7 |
108.6 |
15.974 |
32 |
64.0 |
68.1 |
71.0 |
75 |
169.7 |
31.9227 |
800
n-gram
451
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.48 |
18.90 |
19.83 |
28.71 |
28.9 |
0.999363 |
64 |
115.9 |
116 |
166 |
224 |
383 |
63.7303 |
128 |
200 |
195 |
220 |
412 |
727 |
127.027 |
256 |
434.0 |
349.6 |
851 |
1490 |
2193 |
247.56 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
640 |
160
n-gram
94
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.07 |
19.10 |
19.60 |
19.71 |
21.44 |
0.999260 |
8 |
37.5 |
37.3 |
38.80 |
39.13 |
48 |
7.987 |
16 |
52.7 |
49.3 |
64.5 |
66.7 |
69.2 |
15.963 |
32 |
71.5 |
78.3 |
81.8 |
83.0 |
89 |
31.9033 |
800
n-gram
483
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.49 |
19.70 |
20.8 |
21.9 |
21.54 |
0.999267 |
64 |
133.8 |
128.9 |
174 |
191 |
201.8 |
63.59 |
128 |
215.0 |
211 |
306 |
349 |
429 |
126.29 |
256 |
366.6 |
371.5 |
510 |
682 |
720 |
249.24 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1350 |
160
n-gram
104
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
18.33 |
18.27 |
18.57 |
18.73 |
21.5 |
0.999293 |
8 |
38.1 |
37.31 |
39.0 |
39.2 |
52.7 |
7.9862 |
16 |
51.4 |
51.2 |
64.7 |
66.0 |
68.5 |
15.9627 |
32 |
61.1 |
68.4 |
70.71 |
71.37 |
75 |
31.9113 |
48 |
87.5 |
94.0 |
98.0 |
99.0 |
112.5 |
47.800 |
800
n-gram
479
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.12 |
19.12 |
19.7 |
20.28 |
20.466 |
0.999300 |
64 |
112 |
114 |
131.5 |
156 |
192.4 |
63.61 |
128 |
177.4 |
174 |
222 |
298 |
329 |
126.59 |
256 |
358.8 |
374 |
464 |
740 |
797 |
249.6 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1770 |
160
none
69
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
21.14 |
20.88 |
21.17 |
21.7 |
29.1 |
0.998043 |
8 |
48.6 |
48.6 |
55 |
61.4 |
85.2 |
7.945 |
16 |
61.1 |
60.3 |
76 |
81 |
120 |
15.854 |
32 |
77.5 |
82.5 |
90.1 |
96 |
169 |
31.5517 |
48 |
114.3 |
118.3 |
127.42 |
141 |
295 |
47.01 |
800
none
297
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
23.98 |
23.14 |
28.65 |
29.07 |
29.6 |
0.997983 |
64 |
161 |
147.5 |
224 |
324 |
415 |
62.18 |
128 |
274 |
250 |
401 |
610 |
781 |
121.347 |
256 |
596 |
477 |
1020 |
1530 |
2280 |
228.7 |
none
# of streams |
Throughput (RTFX) |
---|---|
32 |
559 |
160
n-gram
111
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
18.19 |
18.18 |
18.66 |
18.89 |
20.75 |
0.999200 |
8 |
35.63 |
35.37 |
36.6 |
37.2 |
39.1 |
7.9868 |
16 |
51.1 |
46.6 |
64.3 |
65.74 |
69.3 |
15.959 |
32 |
62.8 |
67.6 |
70.2 |
72.3 |
94.3 |
31.885 |
800
n-gram
506
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.21 |
19.29 |
20.3 |
20.9 |
20.92 |
0.999173 |
64 |
118 |
122 |
137.7 |
148.0 |
157.3 |
63.64 |
128 |
174.0 |
179.9 |
213.5 |
219 |
227 |
126.91 |
256 |
295 |
333.2 |
395 |
406 |
452 |
251.81 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1450 |
160
n-gram
100
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
18.43 |
17.79 |
18.30 |
26.88 |
30.55 |
0.998957 |
8 |
37.0 |
36.7 |
36.6 |
52.0 |
95 |
7.9669 |
16 |
48.3 |
45.5 |
62.56 |
66.3 |
146.8 |
15.894 |
32 |
66.4 |
68.4 |
71.7 |
98 |
253.2 |
31.722 |
48 |
97.41 |
92.1 |
102.1 |
156.0 |
438 |
47.391 |
800
n-gram
308
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.9 |
18.96 |
28.35 |
28.7 |
28.7 |
0.999023 |
64 |
161 |
122.6 |
318 |
386 |
461 |
63.120 |
128 |
251 |
201 |
552 |
690 |
810 |
124.56 |
256 |
605 |
384 |
1484 |
2061 |
2316 |
242.61 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
658 |
160
n-gram
97
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.134 |
19.138 |
19.69 |
19.90 |
21.24 |
0.998560 |
8 |
40.4 |
39.3 |
41.5 |
41.8 |
52.7 |
7.971 |
16 |
53.9 |
52.0 |
67.1 |
69.0 |
72.2 |
15.927 |
32 |
66.1 |
73.4 |
78.5 |
79.7 |
82.6 |
31.818 |
800
n-gram
460
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.8 |
19.8 |
20.9 |
20.85 |
20.85 |
0.99841 |
64 |
119 |
124 |
143 |
145 |
153 |
63.34 |
128 |
199 |
212 |
235 |
237 |
242 |
125.55 |
256 |
338 |
372 |
411 |
425 |
434 |
247.8 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1170 |
100
n-gram
216
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
9.8 |
9.4 |
10.35 |
11.26 |
20.78 |
0.999243 |
8 |
15.5 |
14.0 |
17.9 |
19.4 |
58.1 |
7.9810 |
16 |
20.2 |
18.1 |
23.30 |
25.7 |
97.2 |
15.9313 |
32 |
29.6 |
24.9 |
33.1 |
37.3 |
182 |
31.771 |
48 |
39.2 |
31.8 |
42.6 |
50.8 |
265 |
47.518 |
64 |
47.3 |
36.2 |
50.1 |
81.2 |
349 |
63.20 |
800
n-gram
731
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
15.05 |
14.2 |
21.98 |
22.71 |
23.6 |
0.999287 |
64 |
79.6 |
57.4 |
177 |
303 |
393 |
63.187 |
128 |
124.4 |
77.9 |
304 |
529 |
713 |
125.03 |
256 |
218 |
112.6 |
568 |
1009 |
1372 |
245.46 |
384 |
320.6 |
157 |
1027 |
1479 |
2114 |
360.8 |
512 |
493 |
202.1 |
1600 |
2320 |
3160 |
467.6 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
2246 |
100
n-gram
112
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.00 |
18.66 |
19.57 |
20.63 |
30.17 |
0.99898 |
8 |
23.18 |
22.9 |
24.19 |
25.5 |
57.6 |
7.9774 |
16 |
34.0 |
28.5 |
40.0 |
41.5 |
93.1 |
15.926 |
32 |
46.19 |
43.9 |
51.8 |
55.2 |
160.6 |
31.799 |
48 |
57.7 |
52.9 |
59.4 |
64.02 |
260 |
47.572 |
64 |
74.5 |
64.4 |
71.7 |
79.5 |
439 |
63.271 |
800
n-gram
519
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
23.6 |
22.54 |
29.2 |
33.65 |
34.4 |
0.999050 |
64 |
121.0 |
103.2 |
216.2 |
331 |
421 |
63.106 |
128 |
178 |
137 |
329.2 |
542 |
712 |
125.14 |
256 |
324 |
234 |
676 |
1188 |
1556 |
244.9 |
384 |
582 |
343 |
1560 |
2240 |
3180 |
357.2 |
512 |
1059 |
454 |
3350 |
4240 |
6500 |
455.5 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1770 |
160
n-gram
133
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
11.53 |
11.38 |
11.63 |
11.83 |
18.9 |
0.999327 |
8 |
17.9 |
17.3 |
18.90 |
20.91 |
38.2 |
7.98587 |
16 |
29.2 |
28.6 |
32.4 |
33.0 |
67.3 |
15.948 |
32 |
49.3 |
48.0 |
52.0 |
53.7 |
122.6 |
31.797 |
48 |
67.6 |
65.1 |
70.2 |
72.3 |
175.3 |
47.555 |
64 |
81.8 |
78.1 |
85.9 |
89.6 |
265 |
63.237 |
800
n-gram
607
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.18 |
11.78 |
12.19 |
17.08 |
17.54 |
0.999417 |
64 |
82.7 |
76.4 |
91.0 |
143.3 |
277 |
63.381 |
128 |
152 |
148 |
171 |
277 |
539.6 |
125.563 |
256 |
292.3 |
280 |
318 |
553 |
1298 |
246.53 |
384 |
433 |
404 |
457 |
838 |
2280 |
363.2 |
512 |
612 |
525 |
620 |
1468 |
4660 |
475.5 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
2410 |
160
n-gram
152
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
10.51 |
10.58 |
10.92 |
10.99 |
12.39 |
0.999623 |
8 |
17.4 |
17.1 |
18.8 |
19.3 |
21.2 |
7.99477 |
16 |
28.2 |
27.8 |
31.7 |
34.5 |
36.1 |
15.9823 |
32 |
42.8 |
42.4 |
47.7 |
50.0 |
54.2 |
31.9517 |
48 |
53.9 |
54.0 |
57.8 |
59.0 |
64 |
47.902 |
64 |
65.9 |
66.2 |
71.6 |
73.1 |
76.7 |
63.836 |
800
n-gram
691
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
11.52 |
11.66 |
12.43 |
12.88 |
12.833 |
0.999607 |
64 |
79 |
79 |
90 |
93 |
96 |
63.8047 |
128 |
140.5 |
147.3 |
167 |
173 |
187 |
127.27 |
256 |
255.5 |
278.2 |
306 |
314 |
329 |
253.29 |
384 |
378 |
413 |
444 |
450 |
471 |
378.21 |
512 |
499 |
543 |
640 |
690 |
889 |
499.6 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1260 |
160
n-gram
107
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
10.051 |
10.072 |
10.290 |
10.36 |
11.47 |
0.999583 |
8 |
20.5 |
19.19 |
23.01 |
23.8 |
24.0 |
7.9929 |
16 |
26.0 |
25.2 |
32.86 |
33.4 |
34.9 |
15.980 |
32 |
53.4 |
54.8 |
61 |
58.7 |
61.7 |
31.929 |
48 |
69.2 |
77.8 |
83.9 |
85.4 |
88.6 |
47.842 |
64 |
92.3 |
101.9 |
110.4 |
113.5 |
128 |
63.69 |
800
n-gram
508
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.46 |
12.482 |
13.15 |
13.52 |
13.62 |
0.999507 |
64 |
99.7 |
108.0 |
120.6 |
127.2 |
158.2 |
63.686 |
128 |
203.5 |
209.0 |
304 |
322 |
386 |
126.44 |
256 |
362 |
372 |
607 |
638 |
657 |
249.77 |
384 |
529 |
531.5 |
760 |
1009 |
1320 |
369.51 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
965.5 |
160
n-gram
90
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.09 |
12.05 |
12.25 |
12.33 |
14.4 |
0.999517 |
8 |
25.5 |
23.46 |
27.9 |
28.5 |
29.9 |
7.9914 |
16 |
33.4 |
31.5 |
41.0 |
43.9 |
45.9 |
15.976 |
32 |
66.8 |
66.3 |
73 |
75.4 |
81.2 |
31.919 |
48 |
98.3 |
97.7 |
103 |
105.5 |
125.8 |
47.811 |
64 |
121 |
123.6 |
131 |
135 |
202 |
63.59 |
800
n-gram
408
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.08 |
12.01 |
12.535 |
12.90 |
13.1 |
0.999537 |
64 |
134 |
131 |
168.8 |
187 |
245 |
63.60 |
128 |
234 |
234.7 |
330 |
398 |
422 |
126.26 |
256 |
420 |
441 |
510 |
861 |
940 |
248.96 |
384 |
731 |
673 |
1380 |
1552 |
1950 |
367.44 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
921 |
160
n-gram
89
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.12 |
13.16 |
18.32 |
28.56 |
36.09 |
1.000 |
8 |
33.7 |
33.0 |
39.0 |
48.8 |
74.8 |
7.98993 |
16 |
50.6 |
50.2 |
56.9 |
72.4 |
91 |
15.9707 |
32 |
95.5 |
95.5 |
109.8 |
123 |
140 |
31.886 |
48 |
123.0 |
124.2 |
142.5 |
152.7 |
183.7 |
47.7647 |
64 |
167 |
159 |
223 |
261 |
309 |
63.533 |
800
n-gram
401
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
21.19 |
18.35 |
35.6 |
36.2 |
37.0 |
0.999527 |
64 |
146 |
144.5 |
191 |
201 |
213 |
63.67 |
128 |
258 |
261 |
328 |
344 |
349 |
126.73 |
256 |
460 |
455 |
640 |
750 |
931 |
249.5 |
384 |
790 |
695 |
1480 |
1690 |
1800 |
367.5 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
765.6 |
160
n-gram
89
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
11.00 |
10.99 |
11.29 |
11.41 |
12.214 |
0.999503 |
8 |
23.5 |
22.97 |
25.6 |
25.2 |
29.7 |
7.9911 |
16 |
37.5 |
37.0 |
44.8 |
46.3 |
47.5 |
15.9747 |
32 |
67.71 |
67.0 |
74.2 |
75.9 |
97.4 |
31.884 |
48 |
97.2 |
96.0 |
100.2 |
104 |
157.7 |
47.703 |
64 |
125.8 |
123.6 |
129.7 |
134.7 |
202.9 |
63.489 |
800
n-gram
396
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.07 |
12.13 |
12.68 |
12.78 |
12.746 |
0.999493 |
64 |
110.8 |
130 |
142.0 |
145 |
150 |
63.646 |
128 |
220 |
240.3 |
258.0 |
264 |
272 |
126.71 |
256 |
423 |
468 |
502 |
609 |
606 |
250.07 |
384 |
727 |
708 |
1170 |
1420 |
1750 |
368.2 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
875.4 |
160
83
85
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
13.988 |
13.742 |
14.08 |
14.23 |
27.0 |
0.999113 |
n-gram |
8 |
27.3 |
26.00 |
28.0 |
31.2 |
57.7 |
7.9810 |
n-gram |
16 |
39.3 |
36.99 |
45.7 |
46.6 |
100.7 |
15.93 |
n-gram |
32 |
69 |
72.3 |
76.7 |
77.8 |
203 |
31.741 |
n-gram |
48 |
102.1 |
106.2 |
111.1 |
118.3 |
301 |
47.445 |
none |
1 |
12.13 |
11.94 |
12.08 |
12.14 |
18.8 |
0.999367 |
none |
8 |
25.7 |
25.19 |
25.9 |
26.4 |
49.4 |
7.9830 |
none |
16 |
36.3 |
33.9 |
42.6 |
43.6 |
71.1 |
15.938 |
none |
32 |
65.1 |
70.9 |
74.1 |
74.9 |
175 |
31.7493 |
none |
48 |
96 |
102.5 |
106.34 |
109.1 |
292.4 |
47.463 |
none |
64 |
132.1 |
133.00 |
141.4 |
158 |
424 |
63.0483 |
800
365
376
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
15.66 |
15.22 |
15.81 |
23.53 |
25.0 |
0.99922 |
n-gram |
64 |
137 |
137 |
156 |
212 |
345 |
63.270 |
n-gram |
128 |
242 |
243 |
272 |
403 |
685.5 |
125.107 |
n-gram |
256 |
436 |
439 |
507 |
777 |
2181 |
244.75 |
none |
1 |
13.15 |
12.78 |
13.23 |
17.20 |
18.0 |
0.999423 |
none |
64 |
131.9 |
133.0 |
140 |
199.8 |
338 |
63.271 |
none |
128 |
224 |
221 |
256 |
384 |
660 |
125.18 |
none |
256 |
426 |
433 |
496 |
771.7 |
2176 |
244.82 |
Language model |
# of streams |
Speaker Diarization |
Throughput (RTFX) |
---|---|---|---|
n-gram |
32 |
False |
907 |
n-gram |
32 |
True |
41.5 |
none |
32 |
False |
916 |
none |
32 |
True |
42.8 |
160
n-gram
84
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.16 |
13.06 |
13.58 |
13.71 |
19.38 |
0.999543 |
8 |
25.2 |
24.2 |
25.1 |
31.0 |
49.4 |
7.9919 |
16 |
34.97 |
32.73 |
42.7 |
43.9 |
77.8 |
15.978 |
32 |
70.2 |
76.5 |
82.3 |
85.7 |
155.8 |
31.9120 |
800
n-gram
382
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.43 |
14.10 |
15.8 |
20.34 |
20.53 |
0.999533 |
64 |
131 |
133 |
144 |
209.0 |
333 |
63.696 |
128 |
222.4 |
227 |
254.9 |
403.4 |
653 |
126.883 |
256 |
512.70 |
434 |
1025 |
1781 |
2561 |
247.16 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
619 |
160
n-gram
80
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.792 |
13.861 |
14.26 |
14.34 |
16.21 |
0.999463 |
8 |
27.3 |
25.9 |
30.0 |
30.7 |
33.5 |
7.9908 |
16 |
37.5 |
36.3 |
45.1 |
46 |
45.4 |
15.9710 |
32 |
72.4 |
80.5 |
88.7 |
90.3 |
93.5 |
31.902 |
800
n-gram
383
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.52 |
14.639 |
15.9 |
16.64 |
17.3 |
0.999467 |
64 |
138 |
140.2 |
176 |
184 |
188.6 |
63.554 |
128 |
277 |
263 |
410 |
442 |
490 |
125.95 |
256 |
501 |
478 |
820 |
971 |
1300 |
247.15 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1220 |
160
n-gram
90
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.526 |
13.50 |
13.81 |
13.91 |
16.16 |
0.999427 |
8 |
25.0 |
24.26 |
25.4 |
24.76 |
28.0 |
7.9910 |
16 |
34.4 |
32.9 |
34.7 |
42.4 |
44.2 |
15.9763 |
32 |
57.5 |
64 |
71.7 |
73.0 |
77 |
31.921 |
48 |
91.4 |
101.7 |
105.0 |
106.4 |
122 |
47.793 |
800
n-gram
392
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.75 |
13.75 |
14.37 |
14.89 |
15.3 |
0.999480 |
64 |
132.6 |
136.7 |
152 |
194 |
209.9 |
63.53 |
128 |
231 |
231 |
288 |
399 |
430 |
126.19 |
256 |
465 |
471 |
568 |
945 |
1080 |
248.19 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1369 |
160
none
65
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
15.42 |
15.27 |
15.52 |
15.78 |
20.5 |
0.99860 |
8 |
31.1 |
30.0 |
40.8 |
41.5 |
52.2 |
7.9671 |
16 |
57.8 |
59.9 |
68.0 |
73.3 |
87 |
15.890 |
32 |
94 |
95 |
111.3 |
119 |
154 |
31.597 |
48 |
138 |
145 |
161 |
171 |
300 |
47.08 |
800
none
283
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
18.13 |
17.74 |
20.22 |
20.6 |
21.0 |
0.998583 |
64 |
155 |
152 |
181.4 |
261 |
325 |
62.38 |
128 |
284 |
272.6 |
364 |
521 |
654 |
122.46 |
256 |
681 |
535 |
1269 |
1650 |
2380 |
230.6 |
none
# of streams |
Throughput (RTFX) |
---|---|
32 |
631 |
160
n-gram
85
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.45 |
13.47 |
14.04 |
14.26 |
16.30 |
0.999350 |
8 |
25.5 |
25.2 |
26.2 |
26.6 |
28.1 |
7.9893 |
16 |
35.2 |
33.6 |
42.9 |
43.7 |
45.5 |
15.972 |
32 |
62.8 |
70.0 |
74.9 |
76.2 |
96 |
31.880 |
800
n-gram
414
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.63 |
14.8 |
15.84 |
16.28 |
16.82 |
0.999367 |
64 |
123.8 |
134.2 |
145.3 |
148.8 |
154 |
63.633 |
128 |
220 |
244 |
260.6 |
265.7 |
275 |
126.70 |
256 |
391 |
431 |
491 |
576 |
604.8 |
250.30 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1250 |
160
n-gram
82
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.007 |
12.745 |
13.28 |
13.71 |
19.59 |
0.999323 |
8 |
25.7 |
24.37 |
26.6 |
33.0 |
55.7 |
7.9822 |
16 |
38.3 |
34.7 |
45.0 |
47.5 |
93.3 |
15.936 |
32 |
67.4 |
73 |
78.2 |
92.0 |
209.3 |
31.740 |
48 |
111 |
107.2 |
120 |
188 |
449 |
47.426 |
800
n-gram
292
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
15.00 |
13.88 |
20.13 |
20.35 |
20.65 |
0.999290 |
64 |
151 |
135 |
264 |
316 |
357.3 |
63.237 |
128 |
270 |
239 |
517 |
622 |
706 |
124.98 |
256 |
672 |
484 |
1513 |
2157 |
2353 |
244.26 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
666 |
160
n-gram
79
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.858 |
12.84 |
13.37 |
13.57 |
15.04 |
0.999017 |
8 |
26.4 |
25.1 |
32.5 |
33.4 |
36.8 |
7.9784 |
16 |
38.80 |
40.2 |
46.1 |
47.2 |
50.6 |
15.943 |
32 |
67.7 |
75.3 |
79.7 |
80.7 |
85 |
31.8157 |
800
n-gram
371
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.3 |
14.3 |
15.4 |
15.23 |
15.23 |
0.99882 |
64 |
130 |
138.5 |
149.2 |
152.5 |
154.9 |
63.278 |
128 |
236 |
247 |
269 |
274 |
280 |
125.41 |
256 |
427 |
476 |
532 |
548 |
562 |
246.1 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1093 |
100
n-gram
210
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
7.55 |
7.24 |
8.45 |
9.22 |
15.3 |
0.999410 |
8 |
11.4 |
10.5 |
13.2 |
15.2 |
34.9 |
7.98860 |
16 |
15.3 |
14.29 |
17.5 |
19.6 |
55.7 |
15.9613 |
32 |
22.7 |
21.5 |
25.6 |
28.0 |
95.0 |
31.864 |
48 |
29.2 |
26.1 |
32.24 |
35.4 |
153 |
47.691 |
64 |
36.48 |
31.78 |
39.9 |
44.6 |
207 |
63.461 |
800
n-gram
802
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.0 |
12.5 |
17.6 |
18.0 |
18.9 |
0.99943 |
64 |
63.0 |
48.9 |
107.4 |
189.6 |
257.3 |
63.465 |
128 |
104.4 |
76 |
200 |
367.3 |
502.2 |
125.92 |
256 |
189 |
129.6 |
391 |
728.8 |
1095 |
247.79 |
384 |
290.3 |
182.2 |
713.4 |
1225 |
1794 |
365.84 |
512 |
420.7 |
244 |
1220 |
1853 |
2400 |
480.1 |
768 |
945.0 |
374 |
3077 |
4020 |
6039 |
674.7 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
2277.9 |
100
n-gram
76
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.82 |
12.58 |
13.57 |
14.53 |
21.08 |
0.999317 |
8 |
19.9 |
19.0 |
21.59 |
23.05 |
38.1 |
7.9867 |
16 |
30.4 |
29.3 |
33.5 |
35.6 |
62.1 |
15.954 |
32 |
51.6 |
49.6 |
55.2 |
58.2 |
151 |
31.823 |
48 |
73.2 |
67.3 |
74.1 |
78.3 |
303 |
47.626 |
64 |
121.5 |
87.3 |
98.6 |
302.2 |
995.2 |
63.333 |
800
n-gram
449
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
16.91 |
16.18 |
21.5 |
23.9 |
24.5 |
0.999343 |
64 |
109.4 |
94.5 |
161.4 |
243 |
311 |
63.375 |
128 |
202 |
180.1 |
300.0 |
467 |
601 |
125.53 |
256 |
393 |
335 |
670 |
1209 |
1522 |
246.21 |
384 |
832 |
506 |
2176 |
3020 |
4230 |
357.76 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1490 |
160
n-gram
148
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.8 |
12.6 |
12.8 |
12.9 |
19.95 |
0.99928 |
8 |
20.67 |
19.8 |
21.20 |
21.57 |
46.6 |
7.98250 |
16 |
29.1 |
28.0 |
33.4 |
34.8 |
73 |
15.9390 |
32 |
47.6 |
45.8 |
49.5 |
47.7 |
128 |
31.789 |
48 |
64.7 |
62.1 |
67 |
69 |
171.1 |
47.54 |
64 |
77.0 |
73.3 |
78.7 |
81.6 |
265 |
63.203 |
800
n-gram
593
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.7 |
13.2 |
13.6 |
19.3 |
20.2 |
0.999357 |
64 |
85 |
77 |
86.8 |
156 |
303 |
63.332 |
128 |
154.6 |
138.9 |
152 |
292.0 |
579 |
125.42 |
256 |
281 |
254 |
287 |
559 |
1297 |
246.13 |
384 |
416.0 |
369 |
423.3 |
905 |
2310 |
362.0 |
512 |
573 |
487 |
588 |
1453 |
4210 |
473.1 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
2415 |
160
n-gram
158
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.39 |
12.5 |
12.8 |
12.8 |
14.3 |
0.999553 |
8 |
19.9 |
19.70 |
21.0 |
21.5 |
25.0 |
7.99427 |
16 |
29.2 |
28.4 |
33.8 |
34.5 |
36.7 |
15.9817 |
32 |
48.71 |
49.14 |
50.73 |
51.39 |
56.0 |
31.9423 |
48 |
65.6 |
66.2 |
70.4 |
71.6 |
76.1 |
47.8860 |
64 |
71.1 |
71.8 |
76.6 |
78.5 |
82.9 |
63.827 |
800
n-gram
736
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.96 |
13.05 |
13.78 |
14.27 |
14.592 |
0.999543 |
64 |
80.8 |
81.1 |
88.1 |
89.9 |
96.4 |
63.815 |
128 |
134.0 |
134.6 |
148.6 |
153 |
162 |
127.370 |
256 |
235 |
243 |
272 |
277.9 |
300 |
253.650 |
384 |
332 |
353 |
398 |
411 |
439 |
378.81 |
512 |
449 |
476 |
554 |
556 |
698 |
501.0 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1040 |
160
n-gram
125
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.23 |
13.233 |
13.51 |
13.63 |
14.67 |
0.999463 |
8 |
24.54 |
24.26 |
25.63 |
25.99 |
32.5 |
7.99203 |
16 |
32.0 |
30.1 |
42.9 |
45.2 |
46.6 |
15.977 |
32 |
64 |
64 |
69.0 |
71.9 |
78 |
31.908 |
48 |
79.2 |
82.5 |
91.5 |
93.1 |
96.0 |
47.830 |
64 |
94.0 |
100.1 |
112.3 |
113.4 |
118.4 |
63.700 |
800
n-gram
547
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.4 |
13.44 |
14.2 |
14.6 |
14.49 |
0.999473 |
64 |
98.6 |
105.3 |
117.9 |
124.5 |
162.3 |
63.686 |
128 |
181.9 |
183.2 |
255.6 |
278 |
321 |
126.660 |
256 |
321 |
331 |
508 |
566 |
606 |
250.28 |
384 |
470 |
475 |
752 |
960 |
1329 |
370.7 |
512 |
776 |
640.4 |
1610 |
1950 |
2260 |
488.1 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
748.5 |
160
n-gram
107
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.4 |
14.4 |
14.6 |
14.7 |
16.6 |
0.999430 |
8 |
30.36 |
30.13 |
31.24 |
31.69 |
42.8 |
7.98947 |
16 |
45.4 |
37.8 |
57.9 |
58.3 |
62 |
15.971 |
32 |
78.3 |
79 |
82.0 |
82.8 |
89.5 |
31.9043 |
48 |
98.1 |
98.5 |
102.2 |
103.7 |
111.1 |
47.819 |
64 |
122.7 |
122.2 |
127.4 |
129.9 |
166 |
63.652 |
800
n-gram
479
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.0 |
13.9 |
14.363 |
15.0 |
15.41 |
0.999473 |
64 |
127.9 |
125.5 |
140 |
157.8 |
188 |
63.663 |
128 |
200.7 |
202.9 |
237 |
328 |
347 |
126.53 |
256 |
347 |
341 |
413 |
733 |
830 |
249.6 |
384 |
551 |
546 |
1000 |
1220 |
1660 |
369.2 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
744 |
160
n-gram
80
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
16.6 |
15.2 |
20.38 |
31.6 |
41.2 |
0.999423 |
8 |
33.30 |
31.06 |
38.8 |
50.7 |
70.1 |
7.99030 |
16 |
47 |
43 |
60.0 |
76 |
113.3 |
15.971 |
32 |
88.4 |
88 |
98.9 |
121 |
156 |
31.893 |
48 |
122.2 |
121.5 |
137.2 |
163 |
211 |
47.762 |
64 |
150.0 |
148.5 |
177 |
194.5 |
280 |
63.56 |
800
n-gram
346
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
23.4 |
20.7 |
36.3 |
38.2 |
38.5 |
0.999433 |
64 |
168.6 |
163 |
231.7 |
238.1 |
250 |
63.609 |
128 |
315 |
311 |
417 |
453 |
550 |
126.41 |
256 |
638 |
612 |
1012 |
1170 |
1290 |
246.47 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
541 |
160
n-gram
109
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.3 |
14.4 |
14.7 |
14.6 |
15.9 |
0.99936 |
8 |
29.7 |
29.4 |
30.82 |
31.23 |
38.5 |
7.98907 |
16 |
40.6 |
36.3 |
55.8 |
56.8 |
57.6 |
15.968 |
32 |
73.8 |
72.9 |
79 |
81.3 |
96.3 |
31.8807 |
48 |
96.6 |
96.6 |
101.6 |
103.7 |
144.9 |
47.725 |
64 |
120.6 |
119.3 |
124.8 |
127.1 |
192 |
63.548 |
800
n-gram
490
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.9 |
14.9 |
15.61 |
15.96 |
16.5 |
0.999353 |
64 |
125.0 |
124.7 |
131.6 |
135 |
140.9 |
63.663 |
128 |
185 |
197 |
216 |
222 |
228 |
126.920 |
256 |
313 |
331 |
396 |
412 |
454 |
251.74 |
384 |
472 |
539 |
670 |
790 |
871 |
372.5 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
707 |
160
100
103
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
19.2 |
18.7 |
19.45 |
19.65 |
37.8 |
0.998757 |
n-gram |
8 |
41.3 |
40.4 |
41.9 |
42.3 |
92.2 |
7.9708 |
n-gram |
16 |
49.7 |
46.63 |
58.7 |
65.7 |
114 |
15.918 |
n-gram |
32 |
80.1 |
82 |
94.1 |
95.1 |
249 |
31.705 |
n-gram |
48 |
111.9 |
118.9 |
125.8 |
129 |
319 |
47.406 |
none |
1 |
17.3 |
16.9 |
17.1 |
17.2 |
29.4 |
0.999037 |
none |
8 |
37.85 |
36.97 |
37.59 |
37.907 |
77.0 |
7.97507 |
none |
16 |
46.1 |
43.146 |
57.6 |
61.9 |
97.26 |
15.927 |
none |
32 |
74.1 |
75.6 |
89.6 |
90.5 |
207 |
31.726 |
none |
48 |
105.2 |
115.6 |
119.5 |
122.08 |
308 |
47.419 |
none |
64 |
137.5 |
140.3 |
148.7 |
156.6 |
432 |
62.985 |
800
405
434
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
20.4 |
20.0 |
20.69 |
28.8 |
29.76 |
0.999043 |
n-gram |
64 |
146 |
148.2 |
162.4 |
224 |
374 |
63.192 |
n-gram |
128 |
244 |
247 |
272 |
409 |
709 |
124.940 |
n-gram |
256 |
415 |
398 |
471 |
770 |
2129 |
244.44 |
none |
1 |
17.3 |
16.8 |
17.1 |
22.0 |
23.04 |
0.999263 |
none |
64 |
132.5 |
138.2 |
151 |
217 |
365 |
63.195 |
none |
128 |
218 |
214 |
262.1 |
395 |
691 |
125.00 |
none |
256 |
379 |
364 |
448 |
746 |
2064 |
244.66 |
none |
384 |
622 |
570 |
659 |
1565 |
4900 |
358.2 |
Language model |
# of streams |
Speaker Diarization |
Throughput (RTFX) |
---|---|---|---|
n-gram |
32 |
False |
895 |
n-gram |
32 |
True |
92.7 |
none |
32 |
False |
870 |
none |
32 |
True |
95.4 |
160
n-gram
101
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
17.4 |
17.4 |
17.8 |
18.0 |
24.4 |
0.99939 |
8 |
39.7 |
38.85 |
40.0 |
40.8 |
67.8 |
7.9888 |
16 |
45.8 |
43.7 |
48.9 |
63.5 |
90.7 |
15.973 |
32 |
76.1 |
77.8 |
92.03 |
94.0 |
219 |
31.9003 |
800
n-gram
422
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
18.5 |
18.1 |
19.395 |
25.3 |
25.5 |
0.999383 |
64 |
140.1 |
142.7 |
155 |
231 |
368 |
63.668 |
128 |
236.8 |
240.4 |
258.8 |
421 |
696 |
126.91 |
256 |
480 |
388 |
971 |
1707 |
2430 |
247.22 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
575 |
160
n-gram
90
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
18.77 |
18.828 |
19.26 |
19.35 |
21.80 |
0.999267 |
8 |
40.714 |
40.21 |
42.44 |
42.94 |
53.6 |
7.98747 |
16 |
53.7 |
50.6 |
68 |
70.6 |
73.7 |
15.966 |
32 |
77.0 |
87.4 |
95.1 |
96.4 |
100.9 |
31.890 |
800
n-gram
408
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.17 |
20.39 |
21.5 |
22.61 |
23.0 |
0.999243 |
64 |
153.2 |
159.5 |
195 |
205 |
248 |
63.497 |
128 |
268.3 |
267.0 |
403 |
423 |
467.8 |
125.99 |
256 |
477 |
462 |
810 |
930 |
1030 |
247.4 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1100 |
160
n-gram
104
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
17.5 |
17.5 |
17.8 |
17.9 |
20.1 |
0.99929 |
8 |
39.55 |
39.06 |
40.4 |
42.0 |
49.7 |
7.9867 |
16 |
48.6 |
47.3 |
65 |
67 |
72 |
15.962 |
32 |
75.4 |
87 |
92.9 |
93.6 |
99 |
31.889 |
48 |
105.2 |
118.9 |
122.87 |
124.4 |
148.3 |
47.738 |
800
n-gram
458
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
18.7 |
18.6 |
19.1 |
20.63 |
22.1 |
0.99929 |
64 |
139.2 |
146.5 |
178.5 |
194.2 |
233.7 |
63.558 |
128 |
234.5 |
242 |
315 |
371 |
431 |
126.33 |
256 |
384 |
395 |
472 |
832 |
967.2 |
248.96 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1528 |
160
none
55
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.0 |
18.8 |
19.1 |
19.4 |
25.0 |
0.99836 |
8 |
49.1 |
46.5 |
62.7 |
65.0 |
77.7 |
7.9499 |
16 |
63.2 |
62.0 |
78.7 |
81.6 |
105 |
15.870 |
32 |
106.6 |
111.10 |
121.9 |
128 |
195 |
31.548 |
48 |
154.7 |
162.3 |
182 |
192 |
317.7 |
47.03 |
800
none
213
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
22.2 |
21.7 |
24.4 |
25.0 |
25.6 |
0.99832 |
64 |
202.8 |
196.8 |
256 |
340 |
418 |
62.11 |
128 |
394.9 |
384 |
527 |
700 |
936 |
120.8 |
none
# of streams |
Throughput (RTFX) |
---|---|
32 |
591 |
160
n-gram
107
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
17.6 |
17.6 |
18.38 |
18.57 |
20.8 |
0.999167 |
8 |
38.21 |
38.03 |
38.81 |
39.7 |
48.29 |
7.98683 |
16 |
52 |
48.6 |
66 |
67.5 |
71.9 |
15.9530 |
32 |
73.0 |
75.0 |
88.9 |
90.1 |
113.9 |
31.845 |
800
n-gram
481
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
18.4 |
18.6 |
19.54 |
20.71 |
21.07 |
0.999 |
64 |
127.8 |
138 |
150.7 |
154.7 |
159 |
63.615 |
128 |
207 |
231 |
249 |
253 |
263 |
126.723 |
256 |
333 |
354 |
444 |
479 |
522 |
251.17 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1231 |
160
n-gram
99
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
17.7 |
17.3 |
17.8 |
23.9 |
26.3 |
0.99908 |
8 |
39.0 |
37.3 |
38.8 |
50.8 |
84 |
7.9726 |
16 |
47.3 |
45.2 |
61.8 |
65.4 |
118 |
15.9183 |
32 |
81.0 |
85.8 |
90 |
110 |
268 |
31.721 |
48 |
118.2 |
116.6 |
127.6 |
221 |
477.5 |
47.393 |
800
n-gram
331
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.5 |
18.3 |
25.1 |
25.2 |
25.4 |
0.99914 |
64 |
158.5 |
141 |
285.3 |
339.4 |
382.7 |
63.172 |
128 |
278 |
241 |
533.0 |
643.8 |
732 |
124.87 |
256 |
607 |
427 |
1423 |
1895 |
2236 |
244.41 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
653 |
160
n-gram
89
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
17.8 |
17.7 |
18.6 |
18.8 |
21.2 |
0.99864 |
8 |
41.2 |
40.7 |
42.420 |
44.2 |
59.6 |
7.97560 |
16 |
53.2 |
51.1 |
59.6 |
72.4 |
75.3 |
15.929 |
32 |
80.6 |
90.3 |
98.4 |
99.5 |
104.0 |
31.775 |
800
n-gram
411
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
18.6 |
18.8 |
19.9 |
20.2 |
20.2 |
0.99847 |
64 |
148.2 |
160.6 |
172.3 |
178 |
184 |
63.157 |
128 |
251 |
268 |
302 |
308 |
316 |
125.32 |
256 |
410 |
448.2 |
494.8 |
514 |
535 |
246.6 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
946 |
100
n-gram
206
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
9.01 |
8.64 |
9.83 |
10.49 |
17.9 |
0.99929 |
8 |
12.12 |
11.13 |
13.1 |
15.1 |
42.9 |
7.9849 |
16 |
16.4 |
14.53 |
19.0 |
20.8 |
66.7 |
15.9530 |
32 |
21.94 |
20.07 |
25.2 |
28.2 |
113.6 |
31.834 |
48 |
29.5 |
27.0 |
32.5 |
37.0 |
167.1 |
47.657 |
64 |
38.51 |
34.2 |
42.2 |
49.0 |
238.9 |
63.416 |
800
n-gram
779
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.47 |
13.68 |
19.86 |
20.41 |
21.7 |
0.999 |
64 |
65.6 |
50.4 |
118 |
209 |
282 |
63.413 |
128 |
108.2 |
76 |
214.0 |
392.9 |
536.5 |
125.780 |
256 |
190.6 |
121 |
397.1 |
752 |
1101 |
247.377 |
384 |
280.9 |
161.1 |
715 |
1215 |
1752 |
364.9 |
512 |
398 |
211 |
1175 |
1780 |
2339 |
479.2 |
768 |
819.3 |
315 |
2676 |
3559 |
5140 |
674.9 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1627 |
100
n-gram
75
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
22.4 |
22.0 |
23.14 |
24.2 |
31.22 |
0.999003 |
8 |
25.90 |
25.04 |
26.61 |
28.8 |
50.5 |
7.9829 |
16 |
35.6 |
33.0 |
42.68 |
44.8 |
71 |
15.9463 |
32 |
59.4 |
57.1 |
62.9 |
67.2 |
171.2 |
31.810 |
48 |
82.7 |
74.2 |
82.0 |
91.1 |
391.9 |
47.606 |
64 |
164 |
96.3 |
157.1 |
597 |
1394 |
63.290 |
800
n-gram
422
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
27.02 |
26.2 |
32.7 |
35.2 |
36.506 |
0.999033 |
64 |
121.1 |
104.0 |
182 |
271 |
343 |
63.314 |
128 |
220.0 |
186.6 |
326.4 |
508.2 |
651.5 |
125.34 |
256 |
437.9 |
356 |
718 |
1263 |
1577 |
245.61 |
384 |
924 |
523 |
2470 |
3480 |
4680 |
354.87 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1360 |
160
n-gram
64
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.76 |
20.18 |
20.33 |
20.39 |
48.0 |
0.998490 |
8 |
41.70 |
41.6 |
42.3 |
42.88 |
120.1 |
7.9537 |
16 |
54.7 |
53.8 |
60.5 |
61.9 |
164 |
15.870 |
32 |
76.81 |
72.0 |
76.18 |
79.8 |
268.0 |
31.618 |
48 |
122.6 |
111.9 |
119.2 |
129.6 |
471 |
47.136 |
800
n-gram
276
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
23.23 |
22.04 |
22.45 |
38.499 |
39.2 |
0.998783 |
64 |
157.6 |
141.9 |
153 |
290 |
567 |
62.776 |
128 |
323.2 |
286.5 |
313 |
572.5 |
1340 |
123.207 |
256 |
753.7 |
584 |
920 |
1710 |
5600 |
236.20 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1185 |
160
n-gram
67
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.904 |
20.864 |
21.53 |
21.70 |
22.86 |
0.999277 |
8 |
41.62 |
41.60 |
42.26 |
42.49 |
44.02 |
7.9884 |
16 |
55.3 |
55.5 |
60.9 |
62.0 |
64.5 |
15.9670 |
32 |
70.6 |
70.7 |
74.0 |
75.0 |
76.9 |
31.9207 |
48 |
109.1 |
110.2 |
114.7 |
116.2 |
119.0 |
47.818 |
64 |
141.9 |
141.5 |
148.4 |
152.3 |
220.3 |
63.6120 |
800
n-gram
326
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
22.25 |
22.42 |
23.02 |
23.59 |
24.1 |
0.999247 |
64 |
144 |
143 |
155 |
151.4 |
154.5 |
63.68 |
128 |
279 |
281.3 |
294.8 |
298.0 |
313 |
126.750 |
256 |
585.0 |
571 |
720 |
780 |
1140 |
249.68 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1008 |
160
n-gram
59
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.28 |
20.29 |
20.49 |
20.53 |
22.3 |
0.99919 |
8 |
45.97 |
45.90 |
46.38 |
46.58 |
48.5 |
7.98573 |
16 |
56.3 |
56.2 |
61.24 |
62.5 |
66 |
15.965 |
32 |
90.3 |
90.8 |
95.2 |
97.3 |
101.2 |
31.8817 |
48 |
130 |
136.6 |
143.2 |
145.6 |
248 |
47.659 |
800
n-gram
274
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.66 |
20.741 |
21.3 |
21.83 |
22.5 |
0.999 |
64 |
169.8 |
172.5 |
253 |
263.4 |
338 |
63.365 |
128 |
338 |
340.2 |
576 |
601 |
664 |
125.15 |
256 |
780 |
678 |
1284 |
1546 |
1894 |
244.1 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
807 |
160
n-gram
47
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
24.78 |
24.73 |
24.90 |
24.94 |
27.00 |
0.999107 |
8 |
55.2 |
55.69 |
58.55 |
58.81 |
62 |
7.9847 |
16 |
58.15 |
57.43 |
62.02 |
63.6 |
71 |
15.9643 |
32 |
120.2 |
119.9 |
122.2 |
123.0 |
169 |
31.8230 |
800
n-gram
216
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
25.60 |
25.52 |
25.89 |
26.81 |
27.8 |
0.999 |
64 |
229 |
226.1 |
368 |
394 |
459 |
63.183 |
128 |
428 |
441.6 |
660 |
828 |
842 |
124.64 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
684.6 |
160
n-gram
51
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
24.27 |
23.03 |
27.93 |
37.9 |
45.7 |
0.999150 |
8 |
54.6 |
52.3 |
60.6 |
70 |
100.4 |
7.98443 |
16 |
68.2 |
65.5 |
76.5 |
96 |
125.9 |
15.9607 |
32 |
117 |
118 |
127.7 |
150 |
182 |
31.8523 |
48 |
217 |
199.5 |
313 |
332 |
343.7 |
47.587 |
800
n-gram
228
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
31.33 |
28.76 |
43.4 |
44.98 |
45.3 |
0.999130 |
64 |
220 |
218.4 |
271 |
284 |
290 |
63.4827 |
128 |
467 |
443.4 |
660 |
755 |
911 |
124.777 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
616 |
160
n-gram
46
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
24.97 |
24.97 |
25.22 |
25.37 |
26.40 |
0.999003 |
8 |
55.6 |
55.3 |
58.78 |
59.25 |
95 |
7.9805 |
16 |
58.2 |
56.50 |
62.5 |
66.0 |
104.9 |
15.9557 |
32 |
120.9 |
119.11 |
121.6 |
122.9 |
202.3 |
31.7667 |
800
n-gram
221
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
25.57 |
25.68 |
26.17 |
26.3 |
27.1 |
0.999003 |
64 |
207 |
223 |
236.9 |
238.8 |
241 |
63.427 |
128 |
400.4 |
438.5 |
505 |
532 |
667 |
125.53 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
672.6 |
160
39
40
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
28.60 |
27.84 |
28.36 |
28.59 |
66.3 |
0.997923 |
n-gram |
8 |
56.8 |
57.1 |
62.5 |
64.3 |
144.8 |
7.949 |
n-gram |
16 |
62.18 |
59.40 |
63.34 |
68.9 |
160.1 |
15.8827 |
n-gram |
32 |
144 |
129.9 |
169 |
224 |
463 |
31.499 |
none |
1 |
32.61 |
31.80 |
32.03 |
32.09 |
69.13 |
0.997853 |
none |
8 |
54.9 |
55.6 |
61.0 |
62.0 |
132.6 |
7.954 |
none |
16 |
59.8 |
56.9 |
60.9 |
71 |
148.7 |
15.883 |
none |
32 |
140 |
126.8 |
166 |
215 |
459 |
31.504 |
800
175
177
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
36.07 |
34.93 |
35.57 |
54.86 |
55.52 |
0.998277 |
n-gram |
64 |
238.7 |
251 |
280 |
408 |
679 |
62.57 |
n-gram |
128 |
514.5 |
500 |
562 |
896 |
2316 |
122.22 |
none |
1 |
34.34 |
33.25 |
33.98 |
48.59 |
49.4 |
0.998477 |
none |
64 |
224 |
217 |
272.0 |
411 |
681 |
62.559 |
none |
128 |
499 |
488 |
545 |
838 |
2285 |
122.303 |
Language model |
# of streams |
Speaker Diarization |
Throughput (RTFX) |
---|---|---|---|
n-gram |
32 |
False |
425 |
n-gram |
32 |
True |
91 |
none |
32 |
False |
424.4 |
none |
32 |
True |
93 |
160
n-gram
39
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
27.53 |
27.05 |
27.55 |
27.74 |
44.83 |
0.999077 |
8 |
52.9 |
53.9 |
61.1 |
63.2 |
99.4 |
7.9869 |
16 |
61.4 |
58.38 |
64 |
73.4 |
146 |
15.9687 |
800
n-gram
174
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
29.00 |
27.87 |
29.0 |
45.26 |
45.9 |
0.999077 |
64 |
231 |
242 |
276 |
429.1 |
685.2 |
63.424 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
295 |
160
n-gram
39
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
34.4 |
34.5 |
35.0 |
35.1 |
37.1 |
0.998697 |
8 |
58.5 |
58.4 |
63.64 |
64.7 |
67 |
7.9823 |
16 |
63.2 |
63.1 |
66.0 |
68.1 |
77.6 |
15.9600 |
800
n-gram
182
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
34.15 |
34.40 |
35.4 |
36.44 |
36.50 |
0.998727 |
64 |
268 |
276 |
430.2 |
463 |
543 |
62.897 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
567 |
160
n-gram
41
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
33.080 |
33.032 |
33.28 |
33.37 |
36.24 |
0.998813 |
8 |
57.7 |
58.0 |
62.6 |
63.58 |
65.9 |
7.9841 |
16 |
59.5 |
58.8 |
62.1 |
64 |
65.937 |
15.9647 |
32 |
121 |
125.8 |
141.2 |
142.6 |
208 |
31.781 |
800
n-gram
189
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
33.94 |
33.883 |
34.283 |
35.70 |
36.1 |
0.998827 |
64 |
238 |
243 |
348 |
453 |
484 |
63.062 |
128 |
505 |
501 |
692 |
834 |
1038 |
124.08 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
623.8 |
160
none
35
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
33.86 |
33.82 |
33.98 |
34.06 |
36.3 |
0.997720 |
8 |
64.0 |
64.3 |
69.1 |
71.9 |
72.0 |
7.9650 |
16 |
71.1 |
72.1 |
82 |
83.2 |
91.5 |
15.915 |
800
none
149
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
30.32 |
30.37 |
30.9 |
31.2 |
32.6 |
0.998077 |
64 |
283 |
304.8 |
326 |
347 |
384 |
62.711 |
none
# of streams |
Throughput (RTFX) |
---|---|
32 |
479 |
160
n-gram
41
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
27.24 |
27.26 |
27.671 |
27.82 |
30.04 |
0.998863 |
8 |
57.13 |
56.6 |
62.6 |
63.9 |
67 |
7.9792 |
16 |
59.2 |
58.4 |
61.8 |
64.1 |
79.2 |
15.952 |
800
n-gram
195
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
27.28 |
27.45 |
28.5 |
29.16 |
29.33 |
0.999 |
64 |
207.6 |
222 |
267 |
272.2 |
278.8 |
63.360 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
570 |
160
n-gram
41
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
33.40 |
32.66 |
33.23 |
34.53 |
59.13 |
0.998237 |
8 |
56.3 |
55.6 |
61.9 |
63.7 |
108.9 |
7.9697 |
16 |
59.5 |
57.6 |
62.0 |
63.9 |
121.52 |
15.9347 |
800
n-gram
200
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
27.28 |
27.15 |
29.09 |
29.14 |
29.7 |
0.99902000000000002 |
64 |
212 |
232 |
268.3 |
275.9 |
280.6 |
63.427 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
591 |
160
n-gram
39
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
28.61 |
28.58 |
29.40 |
29.72 |
31.3 |
0.997877 |
8 |
58.7 |
58.1 |
63.4 |
64.6 |
67 |
7.9662 |
16 |
62.8 |
62.5 |
65.9 |
65.9 |
69.5 |
15.9260 |
800
n-gram
186
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
34.56 |
34.63 |
35.68 |
36.0 |
36.0 |
0.99735 |
64 |
244 |
272.0 |
287 |
290.6 |
298.2 |
62.656 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
522 |
100
n-gram
116
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.06 |
12.44 |
13.60 |
14.5 |
31.1 |
0.998957 |
8 |
18.7 |
17.2 |
19.42 |
21.8 |
114.1 |
7.9646 |
16 |
26.32 |
22.2 |
28.9 |
42.0 |
197 |
15.881 |
32 |
43.0 |
33.6 |
49.04 |
60.1 |
281 |
31.693 |
48 |
57.9 |
44.12 |
55.3 |
150 |
439 |
47.378 |
800
n-gram
394
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.62 |
17.65 |
33.24 |
33.69 |
34.1 |
0.99895000000000012 |
64 |
136.7 |
104.8 |
298 |
468 |
604 |
62.76 |
128 |
209 |
138 |
421 |
754 |
1122 |
123.85 |
256 |
450 |
254 |
1260 |
1909 |
2657 |
239.89 |
384 |
1020 |
384 |
3350 |
4270 |
6360 |
335.8 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1646 |
100
n-gram
37
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
44.0 |
44.77 |
46.02 |
47.2 |
61.2 |
0.99815 |
8 |
41.3 |
43.2 |
47.06 |
48.5 |
86 |
7.97077 |
16 |
46.1 |
44.3 |
49.9 |
50.6 |
156 |
15.9153 |
32 |
235.4 |
108.1 |
366 |
1028 |
1938 |
31.635 |
800
n-gram
213
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
59.53 |
57.36 |
72.16 |
75.36 |
75.9 |
0.998 |
64 |
210.9 |
176.8 |
322 |
486.1 |
620.1 |
62.740 |
128 |
438.4 |
348.0 |
692 |
1250 |
1566 |
123.047 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
730 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
18.72 |
18.36 |
19.04 |
19.35 |
29.43 |
0.99709 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
18.08 |
18.06 |
18.85 |
18.90 |
27.79 |
0.99806 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
18.66 |
18.46 |
19.66 |
21.44 |
29.31 |
0.99785 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
22.97 |
22.75 |
23.17 |
23.66 |
32.15 |
0.99665 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
22.96 |
19.13 |
33.98 |
52.70 |
67.55 |
0.99781 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
24.03 |
23.97 |
25.00 |
27.26 |
34.79 |
0.99603 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
23.20 |
22.71 |
23.37 |
25.56 |
32.51 |
0.99319 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
22.61 |
22.39 |
23.24 |
24.52 |
32.92 |
0.99653 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
24.85 |
23.57 |
33.93 |
33.93 |
33.93 |
0.98106 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
24.05 |
23.72 |
24.70 |
30.45 |
32.28 |
0.99119 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
28.15 |
27.57 |
29.13 |
29.95 |
53.74 |
0.99476 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
26.43 |
25.42 |
26.99 |
34.16 |
35.23 |
0.98929 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
27.39 |
27.40 |
28.70 |
29.37 |
40.57 |
0.99703 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
25.85 |
25.58 |
26.32 |
26.49 |
33.91 |
0.99664 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
24.90 |
24.38 |
24.83 |
24.92 |
47.63 |
0.99703 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
26.27 |
26.22 |
26.99 |
29.67 |
40.16 |
0.99535 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
25.54 |
25.21 |
26.28 |
29.36 |
39.38 |
0.99595 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
26.93 |
26.64 |
28.21 |
31.86 |
37.60 |
0.99424 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
26.43 |
25.42 |
26.99 |
34.16 |
35.23 |
0.98929 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
28.97 |
26.49 |
41.05 |
41.05 |
41.05 |
0.97404 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
26.93 |
26.37 |
28.08 |
33.67 |
33.73 |
0.9888 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
42.42 |
41.56 |
42.24 |
45.14 |
61.24 |
0.99014 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
31.33 |
29.08 |
40.77 |
44.60 |
44.60 |
0.97936 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
24.14 |
23.93 |
24.50 |
24.61 |
33.04 |
0.99651 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
30.93 |
29.76 |
31.87 |
32.47 |
78.11 |
0.99509 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
27.69 |
28.22 |
29.05 |
29.38 |
34.20 |
0.9971 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
#DIV/0! |
#DIV/0! |
#DIV/0! |
#DIV/0! |
#DIV/0! |
#DIV/0! |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
34.66 |
34.43 |
35.24 |
35.52 |
40.60 |
0.99582 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
33.85 |
29.47 |
44.80 |
78.25 |
99.72 |
0.9966 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
34.38 |
34.29 |
35.33 |
38.06 |
40.40 |
0.99528 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
40.24 |
39.88 |
40.67 |
41.44 |
82.15 |
0.99237 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
33.97 |
33.41 |
36.46 |
37.52 |
53.12 |
0.99453 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
36.80 |
35.83 |
45.55 |
45.55 |
45.55 |
0.9678 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
37.67 |
37.86 |
40.60 |
44.43 |
50.78 |
0.98856 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
39.29 |
38.88 |
39.67 |
40.44 |
81.15 |
0.99237 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
37.11 |
37.00 |
40.15 |
41.47 |
47.38 |
0.99633 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
#DIV/0! |
#DIV/0! |
#DIV/0! |
#DIV/0! |
#DIV/0! |
#DIV/0! |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
38.86 |
38.53 |
39.80 |
40.46 |
58.40 |
0.99417 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
37.06 |
36.12 |
37.86 |
39.29 |
73.63 |
0.99476 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
39.13 |
38.73 |
42.52 |
47.65 |
57.74 |
0.99425 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
36.48 |
36.19 |
37.30 |
38.98 |
60.98 |
0.99379 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
37.77 |
37.92 |
39.47 |
39.86 |
45.46 |
0.99174 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
37.97 |
37.12 |
39.06 |
39.29 |
53.79 |
0.9837 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
41.56 |
39.02 |
59.29 |
59.29 |
59.29 |
0.95947 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
38.59 |
37.96 |
39.44 |
45.12 |
51.30 |
0.98552 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
72.95 |
72.68 |
73.34 |
74.49 |
79.19 |
0.98324 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
46.83 |
43.76 |
60.94 |
70.78 |
70.78 |
0.96824 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
34.95 |
34.63 |
35.39 |
36.24 |
47.59 |
0.99588 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
14.05 |
11.14 |
12.10 |
12.56 |
187.08 |
0.99861 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
12.60 |
11.14 |
11.95 |
12.31 |
102.98 |
0.99889 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
12.73 |
11.34 |
12.01 |
12.66 |
91.21 |
0.99863 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
15.35 |
13.25 |
14.07 |
14.28 |
107.14 |
0.99839 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
16.21 |
13.15 |
20.66 |
36.23 |
96.52 |
0.99835 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
14.31 |
12.77 |
13.44 |
14.02 |
100.79 |
0.99831 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
16.81 |
13.19 |
14.22 |
14.44 |
93.98 |
0.99535 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
14.21 |
12.97 |
13.46 |
13.72 |
98.02 |
0.99859 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
20.97 |
13.02 |
93.78 |
93.78 |
93.78 |
0.99036 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
17.23 |
13.62 |
14.98 |
17.27 |
93.84 |
0.99494 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
15.44 |
14.21 |
14.86 |
14.95 |
99.06 |
0.99777 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
14.81 |
13.47 |
14.38 |
14.78 |
102.76 |
0.99858 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
15.52 |
14.16 |
14.79 |
14.99 |
93.19 |
0.99837 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
15.23 |
14.09 |
14.54 |
14.83 |
93.69 |
0.99828 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
16.78 |
15.28 |
16.17 |
16.65 |
96.42 |
0.99817 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
15.51 |
13.97 |
15.09 |
17.28 |
98.82 |
0.99801 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
14.91 |
13.46 |
14.33 |
16.06 |
101.92 |
0.99823 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
17.35 |
14.39 |
15.96 |
17.01 |
99.81 |
0.99678 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
17.64 |
13.65 |
15.34 |
17.64 |
99.27 |
0.9945 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
23.04 |
14.43 |
96.99 |
96.99 |
96.99 |
0.99655 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
19.73 |
13.22 |
14.87 |
17.95 |
155.80 |
0.99482 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
32.71 |
24.32 |
25.75 |
62.01 |
218.96 |
0.99439 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
21.94 |
16.10 |
20.70 |
97.94 |
97.94 |
0.99057 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
16.72 |
13.26 |
14.37 |
14.97 |
192.65 |
0.99846 |
Hardware Specifications#
GPU |
|
---|---|
NVIDIA DGX A100 40 GB |
|
CPU |
|
Model |
AMD EPYC 7742 64-Core Processor |
Thread(s) per core |
2 |
Socket(s) |
2 |
Core(s) per socket |
64 |
NUMA node(s) |
8 |
Frequency boost |
enabled |
CPU max MHz |
2250 |
CPU min MHz |
1500 |
RAM |
|
Model |
Micron DDR4 36ASF8G72PZ-3G2B2 3200MHz |
Configured Memory Speed |
2933 MT/s |
RAM Size |
32x64GB (2048GB Total) |
GPU |
|
---|---|
NVIDIA A30 |
|
CPU |
|
Model |
AMD EPYC 7742 64-Core Processor |
Thread(s) per core |
1 |
Socket(s) |
2 |
Core(s) per socket |
64 |
NUMA node(s) |
2 |
Frequency boost |
disabled |
CPU max MHz |
2250.0000 |
CPU min MHz |
1500.0000 |
RAM |
|
Model |
Samsung DDR4 M393A4K40DB3-CWE 3200MHz |
Configured Memory Speed |
3200 MT/s |
RAM Size |
32x64GB (2048GB Total) |
GPU |
|
---|---|
NVIDIA A10 |
|
CPU |
|
Model |
AMD EPYC 7763 64-Core Processor |
Thread(s) per core |
1 |
Socket(s) |
2 |
Core(s) per socket |
64 |
NUMA node(s) |
8 |
Frequency boost |
enabled |
CPU max MHz |
2450 |
CPU min MHz |
1500 |
RAM |
|
Model |
Samsung DDR4 M393A4K40DB3-CWE 3200 MHz |
Configured Memory Speed |
3200 MT/s |
RAM Size |
16x32GB (512GB Total) |
GPU |
|
---|---|
NVIDIA V100 SXM2 16 GB |
|
CPU |
|
Model |
Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz |
Thread(s) per core |
2 |
Socket(s) |
2 |
Core(s) per socket |
20 |
NUMA node(s) |
2 |
CPU max MHz |
3600 |
CPU min MHz |
1200 |
RAM |
|
Model |
Micron DDR4 36ASF4G72PZ-2G6D1 2667MHz |
Configured Memory Speed |
2133 MT/s |
RAM Size |
16x32GB (512GB Total) |
GPU |
|
---|---|
NVIDIA T4 |
|
CPU |
|
Model |
Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz |
Thread(s) per core |
2 |
Socket(s) |
2 |
Core(s) per socket |
18 |
NUMA node(s) |
2 |
CPU max MHz |
3900 |
CPU min MHz |
1000 |
RAM |
|
Model |
Samsung DDR4 M393A2K43BB1-CTD 2666MHz |
Configured Memory Speed |
2666 MT/s |
RAM Size |
24x16GB (384GB Total) |