Performance
Contents
Performance¶
Evaluation Process¶
This section shows the latency and throughput numbers for streaming and offline configurations of the Riva ASR service on different GPUs. These numbers were captured after the pre-configured ASR pipelines from our Quick Start scripts were deployed. The Jasper, QuartzNet, Conformer and Citrinet-1024 acoustic models were tested.
In streaming mode, the client and the server used audio chunks of the same duration (100ms, 160ms, 800ms depending on
the server configuration). The Riva streaming client riva_streaming_asr_client
,
provided in the Riva client image, was used with the --simulate_realtime
flag to
simulate transcription from a microphone, where each stream was doing 3 iterations
over a sample audio file from the Librispeech dataset (1272-135031-0000.wav).
The command used to measure performance was:
riva_streaming_asr_client \
--chunk_duration_ms=<chunk_duration> \
--simulate_realtime=true \
--automatic_punctuation=true \
--num_parallel_requests=<num_streams> \
--word_time_offsets=true \
--print_transcripts=false \
--interim_results=false \
--num_iterations=<3*num_streams> \
--audio_file=1272-135031-0000.wav \
--output_filename=/tmp/output.json
The riva_streaming_asr_client
returns the following latency measurements:
intermediate latency
: latency of responses returned withis_final == false
final latency
: latency of responses returned withis_final == true
latency
: the overall latency of all returned responses. This is what is tabulated in the tables below.
In offline mode, the command used to measure maximum throughput was:
riva_asr_client \
--automatic_punctuation=true \
--num_parallel_requests=32 \
--word_time_offsets=true \
--print_transcripts=false \
--num_iterations=96 \
--audio_file=5x_1272-135031-0000.wav \
--output_filename=/tmp/output.json
Results¶
Latencies and throughput measurements for streaming and offline configurations are reported in the following tables. Throughput is measured in RTFX (duration of audio generated / computation time).
Note
If the language model is none
, the inference is performed with a greedy decoder. If the language model is n-gram
,
then a beam decoder was used.
For specifications of the hardware on which these measurements were collected, refer to the Hardware Specifications section.
160
251
270
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
10.49 |
10.207 |
10.60 |
11.46 |
16.5 |
0.999433 |
n-gram |
8 |
14.68 |
14.16 |
15.23 |
17.2 |
33.1 |
7.9910 |
n-gram |
16 |
27.2 |
27.3 |
29.7 |
30.6 |
49.4 |
15.9710 |
n-gram |
32 |
39.5 |
38.6 |
41.1 |
42.7 |
78.67 |
31.9133 |
n-gram |
48 |
48.5 |
47.6 |
50.8 |
52.3 |
102 |
47.833 |
n-gram |
64 |
57.8 |
56.5 |
62.3 |
64.8 |
118 |
63.69 |
none |
1 |
9.95 |
9.60 |
9.87 |
12.3 |
14.86 |
0.999543 |
none |
8 |
13.7 |
13.1 |
14.1 |
16.6 |
25.3 |
7.9928 |
none |
16 |
24.17 |
26.4 |
27.6 |
28.1 |
44.1 |
15.9747 |
none |
32 |
37.0 |
36.5 |
37.9 |
39.6 |
66.0 |
31.924 |
none |
48 |
44.4 |
43.7 |
47.7 |
49.9 |
87 |
47.854 |
none |
64 |
50.3 |
49.5 |
55.8 |
59 |
105.7 |
63.766 |
800
1103
1245
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
11.23 |
10.83 |
12.02 |
15.0 |
15.66 |
0.999493 |
n-gram |
64 |
65.3 |
64 |
72.4 |
83.2 |
94.3 |
63.796 |
n-gram |
128 |
101.5 |
100.1 |
115 |
126 |
154 |
127.32 |
n-gram |
256 |
157.9 |
161.5 |
180.7 |
193 |
235 |
253.97 |
n-gram |
384 |
213 |
224 |
252 |
270 |
333.9 |
379.57 |
n-gram |
512 |
266.5 |
290.54 |
324 |
348 |
438 |
504.30 |
n-gram |
768 |
379 |
422 |
482 |
535 |
653 |
750.7 |
none |
1 |
10.32 |
9.94 |
11.6 |
12.81 |
15.4 |
0.99954 |
none |
64 |
55.0 |
53.6 |
62.1 |
70.5 |
81.0 |
63.825 |
none |
128 |
89.7 |
90.7 |
101.2 |
115 |
132.3 |
127.423 |
none |
256 |
141 |
146 |
165 |
177 |
207.7 |
254.15 |
none |
384 |
196 |
210 |
226.4 |
241 |
295 |
380.02 |
none |
512 |
233 |
267 |
295 |
316 |
392 |
505.067 |
none |
768 |
343.0 |
401 |
437 |
466 |
581 |
752.61 |
Language model |
# of streams |
Throughput (RTFX) |
---|---|---|
n-gram |
32 |
4610 |
none |
32 |
5500 |
160
n-gram
250
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
10.30 |
10.211 |
10.505 |
10.75 |
13.5 |
0.999633 |
8 |
14.56 |
14.42 |
15.18 |
16.1 |
20.1 |
7.9948 |
16 |
27.3 |
28.0 |
29.3 |
29.7 |
33.3 |
15.9833 |
32 |
39.1 |
39.4 |
41.2 |
42.0 |
46.1 |
31.9527 |
48 |
48.8 |
48.7 |
52.6 |
53.9 |
62.5 |
47.914 |
64 |
57.1 |
57.5 |
62.8 |
64.8 |
69.3 |
63.873 |
800
n-gram
1192
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
10.694 |
10.70 |
11.6 |
11.53 |
14.18 |
0.999633 |
64 |
60.5 |
59.9 |
66 |
64.6 |
74 |
63.81 |
128 |
99.7 |
98 |
109 |
113 |
119.8 |
127.527 |
256 |
151 |
156 |
177 |
182 |
194 |
254.467 |
384 |
202.0 |
217.6 |
244.2 |
252 |
267 |
380.49 |
512 |
254.3 |
281 |
312 |
320 |
342 |
506.26 |
768 |
378 |
417 |
471 |
493 |
680 |
754.6 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1640 |
160
n-gram
203
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
10.18 |
9.992 |
10.30 |
10.8 |
12.25 |
1.000 |
8 |
16.70 |
16.52 |
17.21 |
17.9 |
20.5 |
7.99453 |
16 |
30.3 |
32.5 |
34.4 |
34.8 |
37.0 |
15.9780 |
32 |
47.4 |
48.58 |
50.1 |
51.3 |
55 |
31.933 |
48 |
57.6 |
58.0 |
63.9 |
65.4 |
71 |
47.890 |
64 |
60.5 |
63.2 |
70.7 |
72.2 |
76.7 |
63.810 |
800
n-gram
915
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
10.44 |
10.46 |
11.3 |
11.7 |
11.84 |
1.000 |
64 |
77 |
78.0 |
90 |
90.9 |
96 |
63.795 |
128 |
105.8 |
113 |
127.3 |
160 |
180.2 |
127.347 |
256 |
185 |
194 |
289 |
312 |
363 |
252.68 |
384 |
275 |
291 |
470 |
517 |
552 |
375.7 |
512 |
348 |
369 |
573 |
690 |
726 |
497.9 |
768 |
573 |
556 |
1016 |
1260 |
1650 |
737.08 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1270 |
160
n-gram
213
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
11.33 |
10.95 |
11.91 |
12.40 |
32.6 |
0.99893 |
8 |
15.87 |
15.31 |
16.8 |
18.0 |
51.7 |
7.9858 |
16 |
25.4 |
23.05 |
31.1 |
32.6 |
93 |
15.948 |
32 |
42.0 |
40.6 |
43.6 |
45.0 |
136 |
31.850 |
48 |
57.0 |
54.1 |
59.9 |
62.9 |
217 |
47.637 |
64 |
66.2 |
62.9 |
70.6 |
75 |
217 |
63.52 |
800
n-gram
772
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
16.3 |
15.4 |
18.16 |
30.4 |
32.1 |
0.99895 |
64 |
81.5 |
75.5 |
99 |
150.1 |
177 |
63.630 |
128 |
137 |
128 |
156.0 |
254 |
381 |
126.37 |
256 |
246 |
191.5 |
305 |
613 |
963 |
248.1 |
384 |
319 |
265 |
536 |
749 |
1234.1 |
368.9 |
512 |
425 |
345 |
843 |
1214 |
1589 |
485.6 |
768 |
702 |
511 |
1570 |
2016 |
2390 |
714.6 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
729 |
160
n-gram
142
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.89 |
12.03 |
14.7 |
24.3 |
29.1 |
0.999560 |
8 |
25.9 |
24.5 |
29.5 |
37.3 |
56.6 |
7.99250 |
16 |
43.9 |
45.3 |
51.4 |
55.8 |
83 |
15.97 |
32 |
79.2 |
79.0 |
86.3 |
97.3 |
120.0 |
31.908 |
48 |
86.92 |
85.1 |
97.3 |
106.8 |
138 |
47.834 |
64 |
108.3 |
107.7 |
119.2 |
129.5 |
164 |
63.723 |
800
n-gram
566
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
18.9 |
16.84 |
29.3 |
30.4 |
30.9 |
0.999557 |
64 |
129 |
124 |
172 |
182.6 |
193 |
63.712 |
128 |
180 |
174 |
234 |
244 |
273 |
127.13 |
256 |
321 |
316 |
420 |
458 |
475 |
252.3 |
384 |
458.0 |
449 |
640 |
762 |
870 |
372.6 |
512 |
684 |
670 |
1011 |
1160 |
1480 |
490.2 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
570 |
160
137
151
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
15.17 |
14.68 |
15.43 |
17.76 |
25.4 |
0.99916 |
n-gram |
8 |
30.7 |
29.81 |
31.4 |
33.8 |
58.6 |
7.983 |
n-gram |
16 |
48.3 |
49.7 |
52.3 |
54.5 |
80.0 |
15.9557 |
n-gram |
32 |
58.0 |
62 |
71.2 |
75.1 |
116 |
31.867 |
n-gram |
48 |
77.7 |
84.8 |
90.2 |
94 |
140 |
47.772 |
n-gram |
64 |
95.1 |
100.7 |
106.9 |
119 |
180 |
63.615 |
none |
1 |
14.138 |
13.74 |
14.120 |
16.5 |
22.4 |
0.999290 |
none |
8 |
29.1 |
27.72 |
28.39 |
30.967 |
53.5 |
7.9852 |
none |
16 |
45.8 |
46.9 |
49.2 |
50.3 |
65.7 |
15.962 |
none |
32 |
52.4 |
57.7 |
66.6 |
69.6 |
102 |
31.890 |
none |
48 |
66.6 |
74.5 |
79 |
81 |
122 |
47.805 |
none |
64 |
82.4 |
91.5 |
94.0 |
101 |
157.2 |
63.6640 |
800
698
742
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
16.2 |
15.58 |
16.30 |
20.7 |
21.4 |
0.999333 |
n-gram |
64 |
106 |
115 |
127.1 |
142 |
161.6 |
63.659 |
n-gram |
128 |
155 |
154 |
176 |
197 |
216 |
127.110 |
n-gram |
256 |
261 |
275 |
303 |
318 |
356 |
252.94 |
n-gram |
384 |
357 |
389 |
433 |
462 |
516 |
377.63 |
n-gram |
512 |
472 |
510 |
577 |
680 |
798 |
499.2 |
none |
1 |
14.47 |
14.02 |
15.9 |
17.32 |
18.3 |
0.99937 |
none |
64 |
91.7 |
101.4 |
111.4 |
125 |
135 |
63.722 |
none |
128 |
138 |
136 |
156 |
173 |
183 |
127.18 |
none |
256 |
228.5 |
250 |
271 |
281 |
324 |
253.22 |
none |
384 |
338 |
375 |
403 |
413 |
464 |
378.15 |
none |
512 |
454 |
497.4 |
553 |
600 |
656 |
501.3 |
Language model |
# of streams |
Throughput (RTFX) |
---|---|---|
n-gram |
32 |
1820 |
none |
32 |
2060 |
160
n-gram
170
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.05 |
13.78 |
14.31 |
14.51 |
20.2 |
0.999513 |
8 |
28.4 |
27.8 |
28.5 |
30.9 |
51.2 |
7.99227 |
16 |
46.1 |
46.56 |
49.39 |
50.26 |
76.7 |
15.9743 |
32 |
52.1 |
56.4 |
65.8 |
67.8 |
98.2 |
31.9353 |
48 |
63.7 |
70 |
74.58 |
76.1 |
125.3 |
47.880 |
64 |
79.8 |
83.57 |
89.5 |
101 |
156 |
63.8017 |
800
n-gram
737
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
15.3 |
14.66 |
15.21 |
20.97 |
21.19 |
0.99944 |
64 |
90.8 |
97.1 |
109 |
141 |
193 |
63.799 |
128 |
126.4 |
119 |
148.9 |
204 |
307 |
127.36 |
256 |
222 |
236.4 |
263 |
389 |
586 |
253.65 |
384 |
345 |
354.5 |
470 |
594 |
960 |
378.11 |
512 |
499.2 |
461.0 |
798 |
1100 |
1965.5 |
497.4 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1150 |
160
n-gram
152
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.36 |
14.25 |
14.73 |
15.14 |
17.00 |
0.999440 |
8 |
29.13 |
29.12 |
29.91 |
30.33 |
36.1 |
7.99073 |
16 |
45.5 |
48.8 |
51.0 |
51.62 |
55 |
15.9687 |
32 |
55.4 |
61.4 |
69.9 |
71.3 |
75.0 |
31.9130 |
48 |
69.96 |
78.7 |
83.9 |
86.0 |
92.1 |
47.840 |
64 |
87.8 |
97.56 |
103.0 |
104.3 |
109.57 |
63.727 |
800
n-gram
681
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
15.0 |
14.90 |
16.1 |
17.0 |
17.5 |
0.999440 |
64 |
102 |
111 |
128 |
131 |
137.7 |
63.69 |
128 |
160 |
151.6 |
211 |
214.5 |
289.5 |
126.85 |
256 |
283 |
272 |
455 |
494 |
564 |
250.9 |
384 |
395 |
393 |
691 |
733 |
799 |
373.0 |
512 |
542 |
516 |
931 |
1087 |
1410 |
493.1 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1740 |
100
296
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
7.14 |
6.79 |
7.96 |
9.00 |
14.0 |
0.999500 |
8 |
9.40 |
9.1 |
10.6 |
11.7 |
22.6 |
7.9926 |
16 |
13.22 |
12.57 |
16.2 |
18.1 |
29.7 |
15.9797 |
32 |
18.6 |
18.1 |
21.8 |
24.1 |
40.5 |
31.9487 |
48 |
24.0 |
23.21 |
27.9 |
30.7 |
54.3 |
47.897 |
64 |
28.0 |
27.6 |
33.5 |
36.7 |
62 |
63.845 |
800
1609
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.4 |
12.08 |
15.7 |
16.1 |
16.03 |
0.999517 |
64 |
42.64 |
40.5 |
60.4 |
64.9 |
72.0 |
63.856 |
128 |
59.5 |
57.6 |
84.1 |
94.3 |
105.3 |
127.550 |
256 |
91.3 |
88 |
137.8 |
153.5 |
175.2 |
254.45 |
384 |
117 |
112 |
179 |
200 |
230 |
380.99 |
512 |
143 |
140 |
212.6 |
255 |
282 |
506.69 |
768 |
190 |
187 |
306.0 |
352 |
410 |
757.06 |
# of streams |
Throughput (RTFX) |
---|---|
32 |
1480 |
100
n-gram
186
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.16 |
12.817 |
14.2 |
15.16 |
19.29 |
0.999387 |
8 |
14.86 |
14.37 |
16.18 |
17.64 |
27.3 |
7.9927 |
16 |
25.7 |
27.0 |
29.94 |
31.7 |
40.6 |
15.98 |
32 |
37.0 |
37.2 |
41.2 |
43.0 |
57.9 |
31.929 |
48 |
42.5 |
43.5 |
48.9 |
51.2 |
70.0 |
47.8803 |
64 |
45.6 |
45.3 |
51.1 |
54.0 |
72.6 |
63.830 |
800
n-gram
1192
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
17.1 |
16.39 |
20.8 |
22.7 |
22.66 |
0.999393 |
64 |
63.5 |
60.7 |
81 |
85.0 |
101 |
63.813 |
128 |
98.0 |
95.44 |
128.3 |
142.3 |
170 |
127.397 |
256 |
136 |
135 |
170.6 |
186.6 |
217.8 |
254.22 |
384 |
183 |
188 |
243 |
263 |
296 |
380.19 |
512 |
229 |
239 |
306 |
341 |
385 |
505.41 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1464 |
160
140
156
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
14.37 |
14.05 |
14.61 |
14.81 |
24.2 |
0.99917 |
n-gram |
8 |
24.15 |
24.4 |
24.74 |
25.2 |
57 |
7.9843 |
n-gram |
16 |
45.6 |
45.3 |
48.4 |
49.1 |
84.4 |
15.9517 |
n-gram |
32 |
60.2 |
59.1 |
67.0 |
69.84 |
116 |
31.873 |
n-gram |
48 |
68.1 |
66.7 |
74.4 |
76.0 |
136 |
47.755 |
n-gram |
64 |
84.4 |
81 |
95.5 |
98 |
191 |
63.592 |
none |
1 |
13.72 |
13.50 |
13.68 |
13.77 |
21.8 |
0.999267 |
none |
8 |
22.2 |
21.23 |
22.1 |
23.0 |
39.8 |
7.9881 |
none |
16 |
40.9 |
41.7 |
44.8 |
45.7 |
75.1 |
15.9587 |
none |
32 |
56.2 |
55.7 |
61.1 |
63.2 |
103.2 |
31.889 |
none |
48 |
59.2 |
59.1 |
64.8 |
65.8 |
119 |
47.806 |
none |
64 |
69.5 |
71.6 |
76.8 |
78.8 |
150 |
63.664 |
800
657
739
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
15.84 |
15.39 |
15.8 |
21.73 |
22.8 |
0.999283 |
n-gram |
64 |
102 |
99 |
121.4 |
131.8 |
155.3 |
63.69 |
n-gram |
128 |
150 |
153.4 |
174 |
195 |
213.5 |
127.08 |
n-gram |
256 |
261 |
278 |
305 |
324 |
384.0 |
252.75 |
n-gram |
384 |
370.0 |
403 |
440 |
464 |
556 |
376.9 |
n-gram |
512 |
509 |
541 |
627 |
790 |
1000 |
497.8 |
none |
1 |
14.54 |
14.10 |
15.2 |
18.89 |
19.04 |
0.999357 |
none |
64 |
91 |
91 |
109.2 |
117 |
123.4 |
63.740 |
none |
128 |
134 |
137 |
152 |
166.0 |
185 |
127.16 |
none |
256 |
247 |
256 |
296 |
313 |
352 |
253.037 |
none |
384 |
326 |
366 |
400 |
426 |
515 |
377.62 |
none |
512 |
415 |
472 |
527 |
620 |
722 |
500.2 |
Language model |
# of streams |
Throughput (RTFX) |
---|---|---|
n-gram |
32 |
3090 |
none |
32 |
3698 |
160
n-gram
150
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.77 |
13.78 |
14.11 |
14.19 |
15.5 |
1.000 |
8 |
23.70 |
24.4 |
26.0 |
31.4 |
37.2 |
7.9918 |
16 |
45.0 |
45.6 |
48.49 |
48.9 |
51.5 |
15.97 |
32 |
60.0 |
59.1 |
68.2 |
69.6 |
72.6 |
31.938 |
48 |
63.4 |
63.2 |
67.1 |
68.7 |
73.5 |
47.887 |
64 |
76.3 |
76.7 |
81.1 |
82.4 |
85.7 |
63.821 |
800
n-gram
707
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.64 |
14.84 |
15.5 |
15.85 |
16.161 |
0.999493 |
64 |
96 |
91.4 |
109.6 |
114.1 |
121.4 |
63.772 |
128 |
139 |
139.3 |
155.9 |
161.6 |
175 |
127.29 |
256 |
249.8 |
260.9 |
287 |
293.8 |
305 |
253.517 |
384 |
350 |
377 |
404.9 |
417 |
439 |
378.68 |
512 |
470 |
501 |
591 |
628 |
900 |
501.00 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1230 |
160
n-gram
139
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.0 |
14.00 |
14.5 |
14.7 |
15.341 |
0.999443 |
8 |
24.8 |
24.2 |
24.50 |
25.6 |
33.4 |
7.9913 |
16 |
46.4 |
48.4 |
49.71 |
50.13 |
53.5 |
15.9697 |
32 |
61.0 |
61.1 |
66.37 |
67.59 |
70.3 |
31.925 |
48 |
71.9 |
71.8 |
75.8 |
77.6 |
82 |
47.863 |
64 |
86.8 |
88.3 |
94.5 |
96.8 |
101.2 |
63.771 |
800
n-gram
634
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.25 |
14.29 |
14.8 |
15.5 |
15.8 |
0.999437 |
64 |
87.2 |
97 |
128 |
133 |
140.5 |
63.768 |
128 |
153 |
159 |
212 |
228.7 |
237.8 |
126.89 |
256 |
274 |
290.4 |
443 |
488 |
519 |
251.09 |
384 |
408 |
425 |
600 |
776 |
817 |
372.6 |
512 |
596 |
564 |
1060 |
1360 |
1660 |
491.4 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1029 |
160
n-gram
127
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
15.71 |
15.43 |
16.38 |
17.1 |
36.8 |
0.99878 |
8 |
26.2 |
25.7 |
28.1 |
30.1 |
79 |
7.974 |
16 |
44.4 |
40.1 |
50.2 |
51.4 |
115.0 |
15.938 |
32 |
65.6 |
62.8 |
72.0 |
74.6 |
169.0 |
31.784 |
48 |
68.8 |
65.3 |
72.0 |
79 |
215.0 |
47.58 |
64 |
86.3 |
82.9 |
90.0 |
97 |
237 |
63.46 |
800
n-gram
513
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.03 |
19.3 |
22.5 |
32.9 |
36.3 |
0.99885 |
64 |
122 |
111 |
185 |
217 |
298 |
63.42 |
128 |
201 |
174.4 |
273 |
422 |
530 |
125.66 |
256 |
344 |
319 |
409 |
719 |
967 |
247.7 |
384 |
506.7 |
440 |
824 |
1221 |
1470 |
365.6 |
512 |
723 |
580 |
1490 |
1730 |
2180 |
481.4 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
679 |
160
n-gram
106
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
16.42 |
15.593 |
18.31 |
27.56 |
32.20 |
0.999413 |
8 |
28.6 |
27.2 |
32.2 |
39.1 |
55.1 |
7.9914 |
16 |
46.5 |
46.8 |
55.6 |
63.3 |
84 |
15.970 |
32 |
85.7 |
84.5 |
93 |
107.5 |
121.3 |
31.900 |
48 |
110 |
109 |
120 |
128 |
145.6 |
47.76 |
64 |
131 |
131 |
143.5 |
150.7 |
176 |
63.64 |
800
n-gram
448
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
22.998 |
21.32 |
32.90 |
36.0 |
35.5 |
0.999 |
64 |
133.5 |
130.1 |
176.2 |
203 |
213 |
63.675 |
128 |
240 |
242 |
287 |
305 |
319 |
126.76 |
256 |
469 |
461 |
628 |
715 |
870 |
249.68 |
384 |
681 |
651 |
1025 |
1200 |
1510 |
368.8 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
542 |
160
84
91
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
20.83 |
20.3213 |
20.86 |
21.9 |
36.4 |
0.99883 |
n-gram |
8 |
44.2 |
42.5 |
43.69 |
54.1 |
88 |
7.9748 |
n-gram |
16 |
55.8 |
55.7 |
67.1 |
73.4 |
90.7 |
15.9540 |
n-gram |
32 |
79 |
85.4 |
89.6 |
94 |
159 |
31.826 |
n-gram |
48 |
111.50 |
118.7 |
128.3 |
138 |
229 |
47.657 |
n-gram |
64 |
143.9 |
146.1 |
164 |
206 |
353.8 |
63.389 |
none |
1 |
19.64 |
19.22 |
19.53 |
19.68 |
32.82 |
0.998937 |
none |
8 |
46.3 |
41.18 |
55.8 |
59.0 |
89.3 |
7.9758 |
none |
16 |
54.5 |
52.2 |
66.1 |
70.8 |
86.4 |
15.945 |
none |
32 |
70.37 |
78.1 |
82.8 |
85 |
142 |
31.844 |
none |
48 |
101.5 |
109.7 |
120.4 |
126.1 |
209 |
47.681 |
none |
64 |
128 |
135.2 |
145 |
164 |
262 |
63.468 |
800
393
431
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
22.57 |
21.98 |
22.9 |
28.7 |
29.6 |
0.999067 |
n-gram |
64 |
149.1 |
157 |
196 |
203 |
211.6 |
63.566 |
n-gram |
128 |
256 |
261 |
298 |
308 |
329 |
126.67 |
n-gram |
256 |
482 |
496 |
548 |
590 |
734 |
251.11 |
n-gram |
384 |
780 |
735 |
1075 |
1540 |
2310 |
370.8 |
none |
1 |
20.42 |
20.06 |
20.67 |
24.75 |
25.8 |
0.999187 |
none |
64 |
127 |
134.4 |
142.7 |
152.7 |
168.2 |
63.656 |
none |
128 |
214.4 |
229 |
257 |
263 |
304 |
126.79 |
none |
256 |
390.5 |
434 |
459 |
486 |
547 |
251.57 |
none |
384 |
643 |
650 |
884 |
1200 |
1573 |
372.27 |
Language model |
# of streams |
Throughput (RTFX) |
---|---|---|
n-gram |
32 |
1200 |
none |
32 |
1228 |
160
n-gram
96
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.98 |
19.76 |
20.16 |
20.24 |
28.8 |
0.999313 |
8 |
46.4 |
41.9 |
58.4 |
60.9 |
77.9 |
7.9839 |
16 |
53.8 |
54.53 |
64.1 |
67.5 |
101 |
15.97 |
32 |
71.0 |
77.5 |
80.5 |
83.7 |
147.9 |
31.9 |
48 |
103.9 |
107.6 |
118.8 |
143 |
258 |
47.813 |
64 |
136.4 |
129.1 |
204 |
275 |
407 |
63.682 |
800
n-gram
425
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
21.77 |
21.18 |
22.12 |
31.1 |
31.34 |
0.999283 |
64 |
131 |
133 |
145.0 |
205 |
295 |
63.695 |
128 |
223.8 |
228 |
246 |
370 |
546 |
126.947 |
256 |
466 |
429 |
713 |
782 |
1440 |
249.92 |
384 |
1013 |
633.0 |
2070 |
2500 |
3330 |
364.433 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
713 |
160
n-gram
83
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.22 |
20.19 |
20.67 |
20.91 |
22.38 |
0.999200 |
8 |
44.2 |
44.3 |
46.4 |
48.3 |
60 |
7.984 |
16 |
56.4 |
57.5 |
65.3 |
65.6 |
71 |
15.961 |
32 |
72.3 |
81.2 |
85.2 |
86.6 |
93 |
31.903 |
48 |
108 |
118 |
129 |
132 |
141 |
47.758 |
64 |
133.9 |
145.0 |
156.5 |
166 |
183 |
63.555 |
800
n-gram
388
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
21.8 |
21.44 |
22.2 |
22.94 |
23.063 |
0.999197 |
64 |
155 |
152.6 |
191 |
211.9 |
241.0 |
63.52 |
128 |
288 |
271 |
427 |
464 |
516 |
125.87 |
256 |
523 |
497 |
850 |
937 |
1240 |
247.48 |
384 |
1230 |
785 |
2410 |
2690 |
2920 |
364.25 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1120 |
100
235
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
9.697 |
9.385 |
10.42 |
11.14 |
17.46 |
0.999350 |
8 |
12.98 |
12.48 |
14.08 |
15.0 |
27.52 |
7.9916 |
16 |
21.62 |
21.23 |
23.4 |
24.8 |
42.8 |
15.9740 |
32 |
29.75 |
28.8 |
32.48 |
34.71 |
63.9 |
31.9227 |
48 |
35.5 |
34.4 |
40.9 |
43.8 |
77.5 |
47.858 |
64 |
40.7 |
39.6 |
49.7 |
53.0 |
90.6 |
63.787 |
800
1377
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.91 |
14.40 |
19.40 |
20.06 |
20.71 |
0.99936 |
64 |
59.5 |
57 |
83.7 |
90.6 |
100 |
63.792 |
128 |
91.5 |
89.4 |
131 |
147.5 |
163 |
127.327 |
256 |
111.9 |
109.0 |
169.3 |
191.3 |
213.4 |
254.03 |
384 |
151 |
149 |
227 |
260 |
295.0 |
380.17 |
512 |
188.9 |
185.7 |
287.3 |
338 |
389 |
505.24 |
768 |
262 |
259.9 |
414 |
494 |
577 |
753.43 |
# of streams |
Throughput (RTFX) |
---|---|
32 |
2277 |
100
n-gram
111
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
18.64 |
18.36 |
19.314 |
20.465 |
26.55 |
0.999143 |
8 |
23.29 |
22.73 |
24.72 |
25.8 |
40.3 |
7.9866 |
16 |
39.4 |
40.9 |
44.0 |
45.4 |
61.8 |
15.9657 |
32 |
46.3 |
45.4 |
52.1 |
54.2 |
71.9 |
31.923 |
48 |
56.1 |
55.9 |
60.9 |
63.6 |
84.9 |
47.852 |
64 |
67.0 |
67.2 |
72.6 |
75.5 |
97.4 |
63.770 |
800
n-gram
773
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
23.6 |
22.7 |
27.79 |
31.74 |
31.90 |
0.999113 |
64 |
100 |
101 |
133 |
143 |
157.9 |
63.713 |
128 |
147 |
138 |
185 |
203 |
207.5 |
127.22 |
256 |
224.6 |
226.2 |
276 |
302 |
328 |
253.25 |
384 |
314 |
326 |
395 |
437 |
479 |
378.26 |
512 |
399 |
420.8 |
522.3 |
600 |
648 |
501.47 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1264 |
160
125
136
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
10.75 |
10.57 |
11.09 |
11.32 |
15.4 |
0.99947 |
n-gram |
8 |
20.3 |
19.6 |
20.18 |
20.76 |
37.3 |
7.9869 |
n-gram |
16 |
37.0 |
37.2 |
40.8 |
41.6 |
66.7 |
15.9630 |
n-gram |
32 |
55.1 |
51.9 |
63.5 |
64.9 |
107 |
31.874 |
n-gram |
48 |
72 |
66.5 |
70.3 |
85.1 |
148 |
47.756 |
n-gram |
64 |
99 |
98 |
108.2 |
110.7 |
216 |
63.566 |
none |
1 |
9.75 |
9.70 |
9.85 |
9.92 |
12.57 |
0.999563 |
none |
8 |
16.58 |
16.4 |
17.0 |
17.4 |
28.0 |
7.99110 |
none |
16 |
30.2 |
30.0 |
32.15 |
32.5 |
51.8 |
15.9700 |
none |
32 |
47.74 |
46.9 |
49.1 |
49.7 |
97.1 |
31.8963 |
none |
48 |
62.2 |
62.3 |
64.9 |
65.9 |
130.6 |
47.786 |
none |
64 |
77 |
78.9 |
83.1 |
84.18 |
173.5 |
63.619 |
800
592
637
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
12.0 |
11.57 |
12.0 |
16.4 |
16.53 |
0.999450 |
n-gram |
64 |
96 |
97 |
107.5 |
111.0 |
128 |
63.728 |
n-gram |
128 |
163 |
165.2 |
189 |
200.7 |
241 |
127.04 |
n-gram |
256 |
294 |
314.3 |
354 |
370 |
442.5 |
252.35 |
n-gram |
384 |
411 |
449.8 |
493 |
553 |
662 |
375.893 |
n-gram |
512 |
604 |
610 |
764.4 |
1030 |
1606 |
495.9 |
none |
1 |
10.62 |
10.34 |
10.915 |
13.6 |
13.90 |
0.999513 |
none |
64 |
72 |
74 |
83.4 |
95 |
113 |
63.765 |
none |
128 |
139.6 |
148.1 |
163 |
175.5 |
208.7 |
127.103 |
none |
256 |
274.8 |
287.7 |
309 |
334 |
408 |
252.62 |
none |
384 |
385 |
426.7 |
451.5 |
497 |
608 |
376.43 |
none |
512 |
550 |
571 |
673 |
903.4 |
1460 |
496.8 |
Language model |
# of streams |
Throughput (RTFX) |
---|---|---|
n-gram |
32 |
2600 |
none |
32 |
2840 |
160
n-gram
128
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
10.17 |
10.178 |
10.46 |
10.54 |
11.891 |
0.999623 |
8 |
20.3 |
18.7 |
24.71 |
25.38 |
26.53 |
7.9938 |
16 |
32.7 |
33.0 |
34.9 |
35.3 |
36.1 |
15.9807 |
32 |
51.3 |
51.6 |
55.0 |
55.1 |
59.5 |
31.9387 |
48 |
67.9 |
67.4 |
71.6 |
71.6 |
74.7 |
47.8793 |
64 |
83.5 |
84.5 |
90.2 |
91.58 |
97 |
63.7973 |
800
n-gram
617
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
11.56 |
11.60 |
12.4 |
12.9 |
12.95 |
0.999587 |
64 |
81.6 |
81.7 |
89.6 |
91.1 |
97.6 |
63.800 |
128 |
154 |
156 |
173 |
178 |
185 |
127.28 |
256 |
290.97 |
304 |
331.1 |
341 |
359 |
253.23 |
384 |
420 |
444 |
482 |
492 |
645 |
377.81 |
512 |
554 |
584.0 |
739 |
802 |
1100 |
498.1 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1088 |
160
n-gram
105
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
10.69 |
10.7 |
11.0 |
11.0 |
12.1 |
0.999550 |
8 |
21.6 |
20.78 |
21.68 |
21.87 |
24.15 |
7.9917 |
16 |
25.8 |
23.70 |
37.7 |
38.3 |
39.3 |
15.9770 |
32 |
56.4 |
57.2 |
59.3 |
59.9 |
62.4 |
31.926 |
48 |
77.8 |
78.5 |
81.7 |
83.3 |
88 |
47.832 |
64 |
103.6 |
105.5 |
113.5 |
116.3 |
125 |
63.720 |
800
n-gram
495
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.53 |
12.61 |
13.35 |
13.7 |
17.70 |
0.999510 |
64 |
107.0 |
106.5 |
122 |
133 |
189.9 |
63.676 |
128 |
190.3 |
196.7 |
289 |
296 |
313 |
126.51 |
256 |
386 |
392 |
672 |
698 |
740 |
249.2 |
384 |
580 |
568 |
1010 |
1190 |
1560 |
368.42 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
930 |
160
n-gram
120
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
11.89 |
11.614 |
12.47 |
12.76 |
31.35 |
0.998947 |
8 |
22.05 |
21.30 |
26.0 |
29.4 |
49.5 |
7.9794 |
16 |
33.2 |
30.0 |
36.3 |
37.5 |
101.4 |
15.936 |
32 |
53.02 |
50.8 |
54.04 |
56.1 |
174 |
31.80 |
48 |
70.11 |
66.6 |
72.0 |
77.6 |
231 |
47.639 |
64 |
88.5 |
85.0 |
91.13 |
95.6 |
244 |
63.493 |
800
n-gram
467
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
16.42 |
15.86 |
18.34 |
29.29 |
32.3 |
0.99899 |
64 |
103.8 |
95.3 |
120.5 |
207.1 |
225.9 |
63.52 |
128 |
218 |
185.0 |
327 |
440 |
620 |
125.46 |
256 |
384 |
334 |
600 |
870 |
1200 |
246.2 |
384 |
560 |
489 |
961 |
1221 |
1544 |
365.9 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
690 |
160
n-gram
87
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.86 |
14.15 |
16.84 |
25.35 |
29.61 |
0.999460 |
8 |
27.9 |
26.7 |
30.4 |
39.4 |
59 |
7.9902 |
16 |
49.1 |
49.9 |
54.8 |
61.0 |
78.2 |
15.972 |
32 |
79.4 |
79.6 |
86.7 |
93.7 |
112 |
31.902 |
48 |
112.4 |
111.1 |
124 |
132 |
153 |
47.771 |
64 |
145 |
145.5 |
168 |
181 |
251 |
63.58 |
800
n-gram
376
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
21.3 |
19.96 |
31.3 |
33.6 |
33.4 |
0.999 |
64 |
150 |
146 |
191 |
199 |
200 |
63.654 |
128 |
284 |
287 |
349 |
362 |
428 |
126.50 |
256 |
540 |
530 |
726 |
860 |
1029 |
248.68 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
586 |
160
71
76
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
14.11 |
13.869 |
14.30 |
14.65 |
22.2 |
0.999277 |
n-gram |
8 |
28.84 |
28.28 |
29.6 |
29.9 |
50.9 |
7.9862 |
n-gram |
16 |
46.49 |
47.8 |
51.2 |
52.2 |
89.8 |
15.950 |
n-gram |
32 |
75.2 |
82.3 |
87.0 |
89.2 |
172 |
31.816 |
n-gram |
48 |
117.2 |
123.8 |
135.3 |
151 |
251 |
47.621 |
n-gram |
64 |
170.6 |
162.7 |
229 |
314 |
450 |
63.33 |
none |
1 |
13.12 |
12.92 |
13.20 |
13.30 |
19.14 |
0.999377 |
none |
8 |
27.41 |
26.7 |
27.78 |
29.3 |
45.4 |
7.9875 |
none |
16 |
39.3 |
37.3 |
45.7 |
46.8 |
70 |
15.9560 |
none |
32 |
64.4 |
72.3 |
77.6 |
79.7 |
156.0 |
31.836 |
none |
48 |
101 |
111 |
116.6 |
124.9 |
213 |
47.677 |
none |
64 |
142.3 |
145.6 |
161 |
218.0 |
344 |
63.379 |
800
343
364
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
16.13 |
15.75 |
16.32 |
21.6 |
21.75 |
0.999287 |
n-gram |
64 |
160 |
168 |
186 |
199 |
224.2 |
63.572 |
n-gram |
128 |
281 |
284 |
318 |
330 |
357 |
126.523 |
n-gram |
256 |
515 |
538.2 |
602 |
730 |
820 |
249.8 |
none |
1 |
13.61 |
13.33 |
13.89 |
16.65 |
17.4975 |
0.999417 |
none |
64 |
131.5 |
143.1 |
151.9 |
167.7 |
185 |
63.629 |
none |
128 |
235 |
252 |
275 |
289 |
327 |
126.63 |
none |
256 |
466 |
502.3 |
553 |
623 |
741 |
250.893 |
Language model |
# of streams |
Throughput (RTFX) |
---|---|---|
n-gram |
32 |
1084 |
none |
32 |
1167 |
160
n-gram
79
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.81 |
12.66 |
13.12 |
13.33 |
18.44 |
0.999543 |
8 |
27.77 |
26.50 |
28.2 |
33.9 |
50.5 |
7.99230 |
16 |
42.8 |
44.5 |
47.4 |
48.3 |
74.7 |
15.9747 |
32 |
68 |
75.0 |
79.2 |
80.8 |
124.7 |
31.917 |
48 |
105 |
113.0 |
118 |
129 |
213 |
47.801 |
64 |
158 |
144.5 |
256 |
348 |
464 |
63.631 |
800
n-gram
362
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.77 |
14.38 |
15.19 |
21.07 |
21.36 |
0.999497 |
64 |
145 |
148.3 |
161.3 |
198 |
266 |
63.659 |
128 |
257 |
260 |
281 |
364 |
504 |
126.813 |
256 |
540 |
502.7 |
825 |
941 |
1460 |
248.83 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
685 |
160
n-gram
75
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.92 |
13.89 |
14.42 |
14.52 |
16.15 |
0.999443 |
8 |
27.7 |
26.97 |
28.5 |
33.0 |
35.9 |
7.9909 |
16 |
40.0 |
38.5 |
48.2 |
49.3 |
53.4 |
15.973 |
32 |
66.3 |
64.5 |
81.4 |
83.0 |
86.3 |
31.897 |
48 |
106 |
117 |
129 |
132 |
141 |
47.727 |
64 |
163 |
173 |
187.1 |
195.2 |
252 |
63.47 |
800
n-gram
340
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.8 |
14.9 |
15.6 |
16.6 |
16.55 |
0.999430 |
64 |
154 |
162.3 |
184.3 |
206 |
216 |
63.475 |
128 |
298.6 |
280.6 |
466 |
498 |
573 |
125.61 |
256 |
600 |
553 |
1057 |
1089 |
1550 |
245.73 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1090 |
100
224
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
7.52 |
7.23 |
8.27 |
8.87 |
14.0 |
0.999453 |
8 |
10.4 |
9.9 |
11.06 |
12.16 |
23.3 |
7.9926 |
16 |
14.2 |
13.8 |
16.2 |
17.6 |
30.6 |
15.9797 |
32 |
20.7 |
20.4 |
24.05 |
26.2 |
46.8 |
31.939 |
48 |
25.8 |
24.6 |
29.9 |
32.2 |
59.9 |
47.881 |
64 |
31.6 |
31.4 |
36.7 |
38.9 |
71 |
63.827 |
800
1212
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.7 |
12.5 |
16.86 |
17.1 |
17.5 |
0.999463 |
64 |
49.5 |
47.3 |
70.9 |
75.0 |
82.0 |
63.839 |
128 |
73.1 |
75 |
109 |
122 |
136 |
127.437 |
256 |
129 |
127.0 |
187 |
215 |
241.4 |
253.92 |
384 |
177.12 |
177.9 |
254.34 |
302.5 |
350.7 |
379.53 |
512 |
225 |
226 |
332 |
401 |
467 |
504.13 |
768 |
318 |
323.8 |
484 |
600 |
696 |
750.30 |
# of streams |
Throughput (RTFX) |
---|---|
32 |
1280 |
100
n-gram
76
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.55 |
12.30 |
13.23 |
14.23 |
19.32 |
0.999367 |
8 |
21.03 |
18.69 |
28.4 |
29.1 |
32.7 |
7.9896 |
16 |
33.94 |
34.0 |
37.0 |
38.5 |
47.8 |
15.9743 |
32 |
52.3 |
51.7 |
56.7 |
59.5 |
81 |
31.9120 |
48 |
72.9 |
72.4 |
78.7 |
81.2 |
104.0 |
47.829 |
64 |
97.4 |
93.4 |
103.2 |
118 |
215 |
63.714 |
800
n-gram
550
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
17.8 |
17.4 |
22.2 |
24.1 |
24.19 |
0.999327 |
64 |
99.9 |
97.5 |
122.1 |
132 |
144 |
63.731 |
128 |
183.0 |
179.5 |
212.3 |
226 |
242 |
127.050 |
256 |
321 |
330 |
378 |
406 |
436 |
252.43 |
384 |
455 |
479 |
561 |
610.5 |
661 |
376.00 |
512 |
658 |
648 |
863 |
1216 |
1620 |
496.26 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1144 |
160
146
154
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
13.66 |
13.44 |
13.62 |
13.730 |
23.70 |
0.999210 |
n-gram |
8 |
21.12 |
20.51 |
21.32 |
21.8 |
45.1 |
7.9868 |
n-gram |
16 |
37.32 |
36.94 |
39.02 |
39.99 |
71 |
15.9583 |
n-gram |
32 |
53.8 |
52.9 |
55.6 |
56.1 |
116.0 |
31.875 |
n-gram |
48 |
69.4 |
67.77 |
72.5 |
74.3 |
140.3 |
47.752 |
n-gram |
64 |
84.3 |
82.3 |
86.7 |
89.6 |
187 |
63.584 |
none |
1 |
12.6 |
12.5 |
12.6 |
12.7 |
18.66 |
0.99933 |
none |
8 |
19.84 |
19.36 |
19.88 |
20.1 |
39.1 |
7.9880 |
none |
16 |
34.8 |
35.60 |
38.4 |
38.8 |
64.7 |
15.9627 |
none |
32 |
50.4 |
49.7 |
52.4 |
53.0 |
97 |
31.891 |
none |
48 |
66.1 |
64.6 |
70.2 |
72.3 |
133 |
47.775 |
none |
64 |
76.1 |
75.4 |
80.7 |
84.0 |
169.3 |
63.625 |
800
649
716
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
14.14 |
13.70 |
14.3 |
19.29 |
20.2 |
0.999357 |
n-gram |
64 |
90 |
88 |
97 |
117 |
137 |
63.716 |
n-gram |
128 |
152 |
149 |
167 |
194 |
236.1 |
126.993 |
n-gram |
256 |
267 |
270 |
295 |
334 |
408 |
252.47 |
n-gram |
384 |
372 |
393 |
436 |
471 |
596 |
376.35 |
n-gram |
512 |
513 |
525 |
654 |
762 |
984 |
497.4 |
none |
1 |
13.09 |
12.687 |
13.4 |
17.40 |
17.77 |
0.999413 |
none |
64 |
80 |
80 |
88.1 |
108.8 |
123.7 |
63.745 |
none |
128 |
142.0 |
139.6 |
152 |
183 |
218 |
127.087 |
none |
256 |
243 |
251.0 |
272 |
323 |
394 |
252.640 |
none |
384 |
343.4 |
369 |
401.2 |
451 |
571.4 |
376.70 |
none |
512 |
459 |
494 |
573 |
719.1 |
920 |
498.79 |
Language model |
# of streams |
Throughput (RTFX) |
---|---|---|
n-gram |
32 |
2570 |
none |
32 |
2980 |
160
n-gram
150
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.28 |
13.31 |
13.62 |
13.72 |
14.97 |
1.000 |
8 |
20.29 |
20.18 |
21.06 |
21.39 |
24.86 |
7.99 |
16 |
35.4 |
35.65 |
37.6 |
38.2 |
41.1 |
15.9783 |
32 |
52.4 |
52.8 |
54.41 |
54.90 |
59.1 |
31.9377 |
48 |
66 |
66.6 |
71 |
72 |
75 |
47.877 |
64 |
78.8 |
79.6 |
85 |
86 |
91 |
63.808 |
800
n-gram
692
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.37 |
13.53 |
14.4 |
14.8 |
15.1 |
0.999527 |
64 |
83.6 |
84 |
90.6 |
92.7 |
104 |
63.805 |
128 |
141.9 |
141.8 |
154.4 |
161 |
167 |
127.323 |
256 |
253 |
259 |
277 |
286 |
306 |
253.46 |
384 |
356 |
379 |
417 |
426 |
446 |
378.17 |
512 |
482 |
506 |
618 |
658 |
940 |
500.3 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
966 |
160
n-gram
120
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.62 |
13.59 |
13.86 |
13.99 |
14.98 |
0.999437 |
8 |
24.67 |
24.52 |
25.58 |
25.8 |
26.7 |
7.9917 |
16 |
46.0 |
47.3 |
49.3 |
49.69 |
52.0 |
15.972 |
32 |
66.0 |
66.5 |
72.0 |
73.4 |
77.1 |
31.916 |
48 |
83.9 |
84.7 |
96.41 |
98.22 |
102.2 |
47.825 |
64 |
98.3 |
97.5 |
118.84 |
120.73 |
130.7 |
63.670 |
800
n-gram
551
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.38 |
14.37 |
15.15 |
15.8 |
15.9 |
0.999420 |
64 |
108 |
108 |
123 |
132 |
132.6 |
63.702 |
128 |
190.3 |
185.6 |
261 |
276 |
342 |
126.61 |
256 |
326 |
334.1 |
545 |
584 |
650 |
250.01 |
384 |
487 |
484 |
818 |
929 |
1295 |
370.4 |
512 |
803 |
647 |
1590 |
2020 |
2380 |
487.75 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
664 |
160
n-gram
128
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.68 |
14.257 |
15.37 |
16.2 |
37.9 |
0.99867 |
8 |
23.7 |
22.3 |
25.09 |
26.41 |
64 |
7.9730 |
16 |
40.3 |
39.1 |
43.1 |
44.1 |
148 |
15.918 |
32 |
58.7 |
56.7 |
60.6 |
63.3 |
192 |
31.787 |
48 |
75.7 |
71.6 |
78.9 |
89.3 |
278.7 |
47.553 |
64 |
93.4 |
87.8 |
98 |
106 |
318 |
63.27 |
800
n-gram
453
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.40 |
19.75 |
22.61 |
36.2 |
39.6 |
0.998783 |
64 |
119.13 |
105.2 |
157 |
201.3 |
290 |
63.39 |
128 |
212 |
179 |
298 |
490 |
675 |
125.27 |
256 |
386 |
315 |
620 |
940 |
1267 |
245.1 |
384 |
561 |
447 |
1240 |
1440 |
2100 |
360.3 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
475 |
160
n-gram
83
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
16.0 |
15.1 |
17.92 |
27.4 |
32.2 |
0.999403 |
8 |
31.8 |
29.9 |
36.2 |
45.2 |
70 |
7.99063 |
16 |
58.5 |
58.7 |
65.9 |
75 |
107.1 |
15.967 |
32 |
90 |
91 |
94.8 |
104.55 |
137 |
31.890 |
48 |
124 |
123.6 |
137 |
149.0 |
183.2 |
47.767 |
64 |
157.8 |
156.8 |
180 |
203 |
278 |
63.589 |
800
n-gram
344
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
22.5 |
20.2 |
33.4 |
34.2 |
35.0 |
0.999400 |
64 |
179.2 |
170 |
242 |
251.1 |
260 |
63.616 |
128 |
298.0 |
292 |
400 |
414.8 |
445 |
126.45 |
256 |
600.1 |
564 |
904 |
1013 |
1150 |
247.6 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
437 |
160
85
86
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
20.79 |
20.340 |
20.66 |
21.02 |
36.77 |
0.998803 |
n-gram |
8 |
43.8 |
42.4 |
44.1 |
44.9 |
85 |
7.9733 |
n-gram |
16 |
64.3 |
66.56 |
76.7 |
78.3 |
103.5 |
15.9343 |
n-gram |
32 |
87.7 |
98.5 |
102.1 |
104.6 |
187 |
31.805 |
n-gram |
48 |
120.69 |
131.8 |
138.2 |
153.0 |
237.3 |
47.6407 |
n-gram |
64 |
160.7 |
164.0 |
188 |
247 |
360 |
63.367 |
none |
1 |
20.04 |
19.59 |
19.84 |
20.05 |
33.48 |
0.998917 |
none |
8 |
41.1 |
39.59 |
40.7 |
41.2 |
77 |
7.977 |
none |
16 |
63 |
63.7 |
69 |
71 |
104.6 |
15.9443 |
none |
32 |
77.6 |
89.7 |
94.0 |
95.1 |
166.5 |
31.8263 |
none |
48 |
111.0 |
123.9 |
127.26 |
131.0 |
213 |
47.6707 |
none |
64 |
144.2 |
151.9 |
166.1 |
196 |
323 |
63.426 |
800
402
415
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
22.13 |
21.64 |
22.14 |
28.6 |
30.1 |
0.999063 |
n-gram |
64 |
157.9 |
169.8 |
181 |
207 |
227 |
63.528 |
n-gram |
128 |
252.0 |
251.0 |
276 |
311 |
345 |
126.62 |
n-gram |
256 |
438 |
463 |
508 |
535 |
624 |
250.70 |
n-gram |
384 |
763 |
712 |
1240 |
1482 |
2210 |
369.8 |
none |
1 |
19.60 |
19.27 |
19.6 |
23.72 |
23.9365 |
0.999 |
none |
64 |
139.3 |
154.2 |
158.9 |
183 |
198.5 |
63.587 |
none |
128 |
229.1 |
222.8 |
245.2 |
271.4 |
306.8 |
126.720 |
none |
256 |
395 |
439 |
480 |
506 |
583 |
251.28 |
none |
384 |
687 |
676.1 |
1030 |
1218 |
1630 |
370.85 |
Language model |
# of streams |
Throughput (RTFX) |
---|---|---|
n-gram |
32 |
1164 |
none |
32 |
1260 |
160
n-gram
99
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.2 |
19.0 |
19.81 |
19.99 |
26.459 |
0.999313 |
8 |
42.1 |
40.97 |
42.91 |
43.8 |
64.5 |
7.98850 |
16 |
61 |
65 |
70 |
73 |
92.2 |
15.966 |
32 |
80.5 |
92.7 |
96.2 |
97.2 |
142.3 |
31.8927 |
48 |
115.4 |
124.4 |
129.0 |
149 |
239 |
47.7893 |
64 |
151.1 |
148.4 |
177 |
272 |
399 |
63.624 |
800
n-gram
432
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.49 |
20.10 |
21.023 |
27.31 |
27.64 |
0.999 |
64 |
142.1 |
150.75 |
160 |
193.8 |
264.2 |
63.653 |
128 |
217.9 |
207 |
247 |
320.0 |
454.4 |
126.923 |
256 |
426 |
424 |
599 |
720 |
1088 |
251.00 |
384 |
836 |
638 |
1560 |
1890 |
2670 |
367.22 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
734 |
160
n-gram
86
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.3 |
19.4 |
19.8 |
19.9 |
22.0 |
0.99922 |
8 |
42.4 |
42.3 |
43.8 |
44.2 |
48.4 |
7.98667 |
16 |
70.4 |
73.2 |
77.7 |
78.60 |
81.0 |
15.9520 |
32 |
83 |
95 |
101.3 |
102.5 |
106.2 |
31.8763 |
48 |
116.1 |
131.7 |
136.9 |
139.4 |
152 |
47.712 |
64 |
151 |
164 |
175 |
183 |
221 |
63.489 |
800
n-gram
387
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.61 |
20.70 |
21.7 |
22.6 |
22.63 |
0.999 |
64 |
158.5 |
166.6 |
210 |
232 |
273 |
63.469 |
128 |
272.0 |
255.0 |
406 |
441 |
514.5 |
125.82 |
256 |
520 |
477 |
919 |
1070 |
1240 |
246.79 |
384 |
1070 |
743 |
2290 |
2378 |
3040 |
364.2 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1074 |
100
221
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
9.259 |
8.89 |
10.04 |
10.76 |
18.3 |
0.999313 |
8 |
11.8 |
10.97 |
12.9 |
13.58 |
29.5 |
7.9917 |
16 |
16.95 |
15.9 |
20.03 |
21.6 |
40.0 |
15.9737 |
32 |
23.3 |
23.6 |
27.9 |
30.2 |
60.4 |
31.924 |
48 |
28.0 |
28.5 |
32.7 |
35.2 |
73.3 |
47.859 |
64 |
34.3 |
34.6 |
40.8 |
45.4 |
90.6 |
63.767 |
800
1136
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.57 |
13.85 |
20.13 |
20.36 |
21.0 |
0.999373 |
64 |
56.7 |
53.7 |
88 |
97 |
108 |
63.786 |
128 |
79.1 |
72.5 |
125.2 |
136 |
151 |
127.35 |
256 |
121 |
112 |
186 |
211 |
238.2 |
253.89 |
384 |
158.2 |
148.1 |
256 |
288 |
335.9 |
379.54 |
512 |
194 |
181 |
322 |
373 |
435 |
504.20 |
768 |
265 |
249 |
448 |
533 |
637 |
751.04 |
# of streams |
Throughput (RTFX) |
---|---|
32 |
1020 |
100
n-gram
75
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
22.1 |
21.9 |
22.97 |
24.0 |
30.45 |
0.999007 |
8 |
25.6 |
24.89 |
26.7 |
27.967 |
41.3 |
7.9879 |
16 |
40.7 |
42.5 |
46.4 |
48.5 |
59.8 |
15.965 |
32 |
64.3 |
64.9 |
74.6 |
76.48 |
96.5 |
31.897 |
48 |
80.3 |
81.4 |
86.9 |
89.9 |
132 |
47.805 |
64 |
134.0 |
117.2 |
205 |
221 |
375 |
63.61 |
800
n-gram
526
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
27.19 |
26.72 |
33.9 |
35.7 |
37.1 |
0.999037 |
64 |
110.4 |
107.7 |
137.0 |
147.0 |
161.3 |
63.692 |
128 |
194.1 |
188 |
235 |
253 |
281 |
126.930 |
256 |
334 |
335 |
402 |
428.9 |
470 |
252.17 |
384 |
466 |
480 |
591 |
643 |
693 |
375.42 |
512 |
668 |
642.1 |
942 |
1457 |
1780 |
495.05 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1126 |
160
62
64
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
23.99 |
23.49 |
23.83 |
23.95 |
47.6 |
0.99848 |
n-gram |
8 |
50.6 |
49.19 |
51.71 |
52.32 |
117 |
7.9691 |
n-gram |
16 |
60.7 |
59.4 |
65.9 |
67.3 |
128 |
15.929 |
n-gram |
32 |
92.8 |
89.1 |
93.6 |
95.4 |
223 |
31.773 |
n-gram |
48 |
142.9 |
134.4 |
141 |
190 |
312 |
47.527 |
none |
1 |
22.82 |
22.299 |
22.46 |
22.57 |
41.9 |
0.998653 |
none |
8 |
47.9 |
46.7 |
47.58 |
48.5 |
102.8 |
7.9707 |
none |
16 |
57.1 |
55.9 |
61.8 |
64.4 |
105.9 |
15.934 |
none |
32 |
88.2 |
85.0 |
87.6 |
88.94 |
200 |
31.795 |
none |
48 |
133.7 |
127.0 |
130.1 |
166 |
298 |
47.566 |
800
287
306
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
25.8 |
25.02 |
25.9 |
36.0 |
36.5 |
0.998837 |
n-gram |
64 |
174 |
173 |
189 |
213 |
254 |
63.49 |
n-gram |
128 |
320 |
322 |
346 |
388 |
455 |
126.07 |
n-gram |
256 |
642 |
621 |
806 |
1100 |
1820 |
247.65 |
none |
1 |
24.11 |
23.520 |
23.81 |
31.73 |
32.08 |
0.999 |
none |
64 |
156 |
157 |
168 |
203 |
226 |
63.531 |
none |
128 |
288 |
303 |
314.9 |
361 |
426 |
126.240 |
none |
256 |
574 |
591 |
757 |
920 |
1546.2 |
248.14 |
Language model |
# of streams |
Throughput (RTFX) |
---|---|---|
n-gram |
32 |
1324 |
none |
32 |
1410 |
160
n-gram
63
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
24.69 |
24.92 |
25.49 |
25.57 |
30.5 |
0.99914999999999987 |
8 |
51.16 |
51.12 |
52.793 |
53.17 |
63.8 |
7.98587 |
16 |
60.7 |
60.4 |
66.4 |
68.3 |
73.0 |
15.965 |
32 |
90.1 |
89.8 |
94.8 |
95.9 |
106 |
31.8967 |
48 |
132.45 |
132.00 |
139.0 |
140.8 |
150.9 |
47.755 |
800
n-gram
299
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
26.8 |
26.8 |
28.3 |
30.30 |
30.30 |
0.999147 |
64 |
173 |
173 |
188 |
196 |
206 |
63.607 |
128 |
307 |
315 |
338 |
345 |
363 |
126.55 |
256 |
599 |
613.2 |
810 |
855 |
1130 |
248.57 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
830 |
160
n-gram
59
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
22.93 |
23.03 |
23.43 |
23.58 |
26.88 |
0.999107 |
8 |
51.53 |
51.24 |
53.41 |
53.9 |
59 |
7.98397 |
16 |
62.9 |
62.4 |
67.5 |
69.4 |
74.8 |
15.959 |
32 |
102.31 |
102.1 |
107.32 |
109.4 |
121.9 |
31.860 |
48 |
148.9 |
149.94 |
158.9 |
167.1 |
211 |
47.612 |
800
n-gram
266
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
23.9 |
23.9 |
24.6 |
27.6 |
27.9 |
0.999110 |
64 |
188 |
170.4 |
276 |
305 |
345 |
63.305 |
128 |
340 |
343 |
597 |
620 |
682 |
125.04 |
256 |
830 |
671 |
1500 |
1705 |
2048 |
243.80 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
734 |
160
n-gram
58
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
29.24 |
28.89 |
31.4 |
32.5 |
69.7 |
0.99770 |
8 |
59.54 |
58.14 |
64.2 |
65.9 |
155.3 |
7.955 |
16 |
69.9 |
68.6 |
77.8 |
80.6 |
203 |
15.891 |
32 |
100.0 |
97.9 |
107.0 |
110.3 |
232.6 |
31.698 |
48 |
153.9 |
142.6 |
156.2 |
188.4 |
459 |
47.30 |
800
n-gram
240
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
40.0 |
40.6 |
45.1 |
63.1 |
64.6 |
0.99791 |
64 |
237 |
201 |
348 |
464 |
545 |
62.95 |
128 |
421 |
373 |
610 |
950 |
1200 |
123.70 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
510 |
160
n-gram
48
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
30.11 |
28.81 |
37.26 |
54.4 |
61.5 |
0.999010 |
8 |
66.60 |
61.9 |
80.0 |
94.6 |
133 |
7.98123 |
16 |
82.8 |
78.8 |
101.0 |
118.6 |
155 |
15.9520 |
32 |
137.1 |
134.5 |
155.8 |
176 |
202 |
31.8243 |
800
n-gram
212
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
43.2 |
43.5 |
62.1 |
63.5 |
67 |
0.999013 |
64 |
269 |
269 |
325 |
349 |
362 |
63.36 |
128 |
492 |
487 |
630 |
681 |
810 |
125.00 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
447 |
160
37
37
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
53.82 |
53.4 |
54.46 |
55.1 |
106 |
0.99668 |
n-gram |
8 |
68.2 |
66.5 |
71.5 |
72.93 |
141 |
7.9630 |
n-gram |
16 |
102.7 |
91.5 |
125 |
135 |
206 |
15.896 |
n-gram |
32 |
305 |
269 |
433 |
501 |
710 |
31.55 |
none |
1 |
51.40 |
50.74 |
50.882 |
51.3 |
96.9 |
0.996957 |
none |
8 |
65 |
62.9 |
66.0 |
70 |
122.8 |
7.9679 |
none |
16 |
96.9 |
84.3 |
121 |
130.1 |
188 |
15.902 |
none |
32 |
252 |
238 |
337 |
436 |
564 |
31.605 |
800
173
180
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
57.2 |
56.39 |
57.9 |
71.13 |
71.44 |
0.997767 |
n-gram |
64 |
309.6 |
328 |
349 |
381 |
426 |
63.170 |
n-gram |
128 |
523 |
526 |
590 |
689 |
798 |
124.78 |
none |
1 |
51.7 |
50.97 |
51.6 |
59.01 |
59.114 |
0.99813000000000007 |
none |
64 |
289 |
307 |
323 |
368 |
400 |
63.231 |
none |
128 |
484 |
426 |
566.8 |
658 |
746 |
125.09 |
Language model |
# of streams |
Throughput (RTFX) |
---|---|---|
n-gram |
32 |
502.5 |
none |
32 |
531 |
160
n-gram
38
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
51.79 |
50.8 |
52.93 |
53.317 |
75.0 |
0.998273 |
8 |
62.3 |
61.2 |
67.7 |
69.8 |
115.4 |
7.9835 |
16 |
95.6 |
84.3 |
118 |
134 |
194 |
15.949 |
32 |
284 |
214 |
544 |
667 |
856 |
31.607 |
800
n-gram
171
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
55.1 |
53.5 |
55.93 |
76.5 |
76.10 |
0.998263 |
64 |
294.0 |
299.1 |
318 |
445 |
609.6 |
63.339 |
128 |
600 |
538.6 |
1067 |
1290 |
1746 |
123.49 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
295 |
160
n-gram
37
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
51.97 |
52.31 |
53.36 |
53.47 |
59.0 |
0.998 |
8 |
65.6 |
65.2 |
69.1 |
70.6 |
72.7 |
7.9799 |
16 |
95 |
86.2 |
114.1 |
116.6 |
121.3 |
15.934 |
32 |
266 |
251 |
385 |
416 |
440 |
31.633 |
800
n-gram
167
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
54.4 |
54.57 |
57 |
60.55 |
60.66 |
0.99801000000000029 |
64 |
317.0 |
318 |
414 |
495 |
515 |
62.784 |
128 |
596 |
560 |
956 |
1110 |
1360 |
122.86 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
488 |
100
118
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
17.05 |
16.24 |
19.48 |
21.3 |
36.0 |
0.99866 |
8 |
25.3 |
23.6 |
27.2 |
29.9 |
63.1 |
7.9821 |
16 |
46.2 |
44.5 |
55.3 |
61.9 |
105.4 |
15.937 |
32 |
47.2 |
46.4 |
59.8 |
66.4 |
113 |
31.866 |
48 |
53.0 |
52.77 |
65.8 |
76 |
127 |
47.770 |
64 |
62.7 |
60.6 |
78.8 |
90.6 |
153 |
63.653 |
800
546
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
39.6 |
39.8 |
48.62 |
49.6 |
52.2 |
0.998430 |
64 |
147 |
143 |
197 |
207.8 |
231.6 |
63.53 |
128 |
189 |
183 |
248 |
282 |
313.25 |
126.73 |
256 |
305 |
296 |
387.6 |
448 |
503 |
251.61 |
384 |
407 |
401 |
535 |
643 |
750 |
374.69 |
512 |
547 |
527 |
755 |
878 |
1100 |
495.2 |
# of streams |
Throughput (RTFX) |
---|---|
32 |
894 |
100
n-gram
37
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
47.3 |
47.2 |
49.80 |
53.9 |
65.85 |
0.99794 |
8 |
47.5 |
47.9 |
52.0 |
55.9 |
79.2 |
7.9790 |
16 |
75.0 |
72.9 |
80.4 |
86.2 |
109.2 |
15.938 |
32 |
242 |
228.1 |
314 |
362 |
576 |
31.761 |
800
n-gram
270
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
69.4 |
69.1 |
78.19 |
84.31 |
86.8 |
0.997623 |
64 |
203.5 |
202.5 |
241.0 |
259 |
281 |
63.468 |
128 |
360 |
356 |
420 |
452 |
482 |
126.11 |
256 |
748 |
685 |
1010.9 |
1524 |
2357 |
247.45 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
747 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
23.96 |
24.47 |
25.14 |
25.95 |
42.33 |
0.99936 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
27.96 |
28.53 |
30.54 |
30.92 |
48.13 |
0.99916 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
25.28 |
24.21 |
29.97 |
30.21 |
40.50 |
0.99914 |
160
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
28.84 |
27.91 |
31.32 |
31.95 |
47.64 |
0.99901 |
Hardware Specifications¶
GPU |
|
---|---|
NVIDIA DGX A100 40 GB |
|
CPU |
|
Model |
AMD EPYC 7742 64-Core Processor |
Thread(s) per core |
2 |
Socket(s) |
2 |
Core(s) per socket |
64 |
NUMA node(s) |
8 |
Frequency boost |
enabled |
CPU max MHz |
2250 |
CPU min MHz |
1500 |
RAM |
|
Model |
Micron DDR4 36ASF8G72PZ-3G2B2 3200MHz |
Configured Memory Speed |
2933 MT/s |
RAM Size |
32x64GB (2048GB Total) |
GPU |
|
---|---|
NVIDIA A30 |
|
CPU |
|
Model |
AMD EPYC 7742 64-Core Processor |
Thread(s) per core |
1 |
Socket(s) |
2 |
Core(s) per socket |
64 |
NUMA node(s) |
2 |
Frequency boost |
disabled |
CPU max MHz |
2250.0000 |
CPU min MHz |
1500.0000 |
RAM |
|
Model |
Samsung DDR4 M393A4K40DB3-CWE 3200MHz |
Configured Memory Speed |
3200 MT/s |
RAM Size |
32x64GB (2048GB Total) |
GPU |
|
---|---|
NVIDIA V100 SXM2 16 GB |
|
CPU |
|
Model |
Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz |
Thread(s) per core |
2 |
Socket(s) |
2 |
Core(s) per socket |
20 |
NUMA node(s) |
2 |
CPU max MHz |
3600 |
CPU min MHz |
1200 |
RAM |
|
Model |
Micron DDR4 36ASF4G72PZ-2G6D1 2667MHz |
Configured Memory Speed |
2133 MT/s |
RAM Size |
16x32GB (512GB Total) |
GPU |
|
---|---|
NVIDIA T4 |
|
CPU |
|
Model |
Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz |
Thread(s) per core |
2 |
Socket(s) |
2 |
Core(s) per socket |
18 |
NUMA node(s) |
2 |
CPU max MHz |
3900 |
CPU min MHz |
1000 |
RAM |
|
Model |
Samsung DDR4 M393A2K43BB1-CTD 2666MHz |
Configured Memory Speed |
2666 MT/s |
RAM Size |
24x16GB (384GB Total) |