Performance
Contents
Performance#
Evaluation Process#
This section shows the latency and throughput numbers for streaming and offline configurations of the Riva ASR service on different GPUs. These numbers were captured after the preconfigured ASR pipelines from our Quick Start scripts were deployed. The Jasper, QuartzNet, Conformer, and Citrinet-1024 acoustic models were tested.
In streaming mode, the client and the server used audio chunks of the same duration (100ms, 160ms, and 800ms depending on the server configuration). The Riva streaming client riva_streaming_asr_client
,
provided in the Riva client image, was used with the --simulate_realtime
flag to
simulate transcription from a microphone, where each stream was doing three iterations
over a sample audio file from the LibriSpeech dataset (1272-135031-0000.wav
).
The command used to measure performance was:
riva_streaming_asr_client \
--chunk_duration_ms=<chunk_duration> \
--simulate_realtime=true \
--automatic_punctuation=true \
--num_parallel_requests=<num_streams> \
--word_time_offsets=true \
--print_transcripts=false \
--interim_results=false \
--num_iterations=<3*num_streams> \
--audio_file=1272-135031-0000.wav \
--output_filename=/tmp/output.json
The riva_streaming_asr_client
returns the following latency measurements:
intermediate latency
: latency of responses returned withis_final == false
final latency
: latency of responses returned withis_final == true
latency
: the overall latency of all returned responses. This is what is tabulated in the following tables.
In offline mode, the command used to measure maximum throughput was:
riva_asr_client \
--automatic_punctuation=true \
--num_parallel_requests=32 \
--word_time_offsets=true \
--print_transcripts=false \
--num_iterations=96 \
--audio_file=1272-135031-0000x5.wav \
--output_filename=/tmp/output.json
Results#
Latencies and throughput measurements for streaming and offline configurations are reported in the following tables. Throughput is measured in RTFX (duration of audio generated / computation time).
Note
If the language model is none
, the inference is performed with a greedy decoder. If the language model is n-gram
,
then a beam decoder was used.
For specifications of the hardware on which these measurements were collected, refer to the Hardware Specifications section.
160
n-gram
246
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
10.47 |
10.09 |
10.61 |
12.1 |
16.33 |
0.99944 |
8 |
15.14 |
14.46 |
15.81 |
19.28 |
34.1 |
7.9872 |
16 |
26.2 |
25.1 |
28.8 |
29.7 |
51.9 |
15.9607 |
32 |
39.1 |
38.5 |
42.6 |
44.0 |
92.8 |
31.847 |
48 |
48.0 |
47.6 |
53.5 |
55.6 |
129 |
47.679 |
64 |
55.4 |
53.9 |
61.1 |
64.1 |
146 |
63.530 |
800
n-gram
977
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
11.4 |
10.79 |
11.417 |
15.209 |
15.44 |
0.999497 |
64 |
64.1 |
59.1 |
68.2 |
112.1 |
204 |
63.556 |
128 |
103 |
98 |
116 |
187 |
345 |
126.39 |
256 |
166.7 |
157.9 |
176.8 |
335 |
665 |
250.14 |
384 |
235 |
225.2 |
264 |
501 |
1150 |
371.18 |
512 |
311 |
294 |
358 |
660 |
1510 |
489.5 |
768 |
492 |
446 |
560 |
1310 |
3250 |
717.1 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
4390 |
160
n-gram
257
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
9.96 |
9.885 |
10.21 |
10.23 |
12.8 |
0.999657 |
8 |
23.5 |
23.0 |
25.1 |
25.7 |
27.5 |
7.99327 |
16 |
26.4 |
25.9 |
29.1 |
29.46 |
32.3 |
15.9837 |
32 |
38.3 |
38.2 |
40.1 |
42.9 |
42.6 |
31.951 |
48 |
47.5 |
47.8 |
52.14 |
53.09 |
57 |
47.913 |
64 |
53.5 |
54.1 |
61.0 |
62.9 |
67.5 |
63.854 |
800
n-gram
1169
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
11.1 |
10.74 |
12.3 |
13.3 |
14.09 |
1.000 |
64 |
59.2 |
59.1 |
67 |
69.8 |
73.1 |
63.847 |
128 |
92 |
96 |
111 |
115 |
123 |
127.51 |
256 |
139.6 |
151.9 |
172 |
183 |
198 |
254.43 |
384 |
189 |
214.0 |
242 |
250 |
261 |
380.6 |
512 |
242.5 |
278 |
318 |
329 |
353 |
506.11 |
768 |
353 |
412 |
481.9 |
500 |
540 |
754.6 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1730 |
160
n-gram
207
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
10.33 |
10.212 |
10.54 |
11.00 |
12.37 |
0.999600 |
8 |
17.10 |
16.82 |
17.86 |
18.58 |
24.56 |
7.9924 |
16 |
33.6 |
31.8 |
43.3 |
45.2 |
47.7 |
15.976 |
32 |
45.5 |
46.6 |
54 |
52.0 |
61 |
31.932 |
48 |
52.1 |
58.1 |
63.8 |
65.7 |
68.7 |
47.8713 |
64 |
60.3 |
68.4 |
74.7 |
76.2 |
79.6 |
63.815 |
800
n-gram
904
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
10.9 |
10.72 |
11.5 |
12.1 |
11.91 |
0.999593 |
64 |
76 |
87.3 |
94 |
94.2 |
100 |
63.775 |
128 |
114 |
118 |
141 |
150.1 |
215 |
127.290 |
256 |
192 |
202 |
278 |
322 |
349 |
252.71 |
384 |
268 |
290 |
399 |
516.4 |
543.6 |
376.04 |
512 |
347 |
373 |
550 |
710 |
749 |
497.6 |
768 |
553 |
559 |
997 |
1120 |
1598 |
735.9 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1281.6 |
160
n-gram
219
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
11.54 |
11.14 |
12.28 |
13.2 |
34.6 |
0.998873 |
8 |
16.868 |
16.11 |
18.2 |
19.5 |
46.3 |
7.980 |
16 |
29.2 |
27.7 |
31.6 |
33.0 |
102 |
15.945 |
32 |
43.70 |
42.09 |
45.8 |
47.6 |
144.2 |
31.835 |
48 |
54.7 |
52.3 |
58.6 |
61.5 |
218 |
47.63 |
64 |
61.2 |
59.1 |
70 |
74 |
222 |
63.45 |
800
n-gram
682
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
16.4 |
15.6 |
18.1 |
33.1 |
34.1 |
0.99883 |
64 |
80.9 |
76 |
104 |
149.7 |
196.9 |
63.56 |
128 |
129.2 |
121 |
165.6 |
340 |
403 |
126.36 |
256 |
204.3 |
191 |
275 |
505 |
720 |
248 |
384 |
290 |
245 |
453 |
1037 |
1120 |
369.9 |
512 |
377.4 |
316.0 |
730 |
1343 |
1410 |
487.2 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
733 |
160
n-gram
153
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.56 |
12.14 |
17.8 |
28.5 |
39.1 |
0.999573 |
8 |
31.4 |
29.0 |
36.903 |
55.4 |
82.0 |
7.9898 |
16 |
45.59 |
43.3 |
54.1 |
70.2 |
96.7 |
15.971 |
32 |
63.6 |
61.8 |
75.36 |
91 |
119 |
31.908 |
48 |
73.4 |
73.9 |
90.0 |
100.1 |
144 |
47.844 |
64 |
88.2 |
93.0 |
105.5 |
123 |
167 |
63.764 |
800
n-gram
563
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.9 |
18.1 |
35.1 |
36.4 |
37.5 |
0.999560 |
64 |
123.3 |
118.4 |
173 |
179.6 |
183.8 |
63.722 |
128 |
178 |
173 |
248 |
261 |
273 |
127.09 |
256 |
302 |
314 |
402 |
416 |
470 |
252.38 |
384 |
470 |
478 |
688 |
746 |
890 |
373.2 |
512 |
621 |
632 |
970 |
1123 |
1243 |
491.6 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
625 |
160
n-gram
170
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
11.73 |
11.598 |
12.01 |
12.6 |
13.40 |
0.999497 |
8 |
28.75 |
29.22 |
30.43 |
30.98 |
33.0 |
7.98967 |
16 |
38.5 |
38.4 |
39.5 |
40.1 |
43.1 |
15.97 |
32 |
54.7 |
54.9 |
57 |
59 |
61.1 |
31.916 |
48 |
62.3 |
62.6 |
67.3 |
69.4 |
90 |
47.833 |
64 |
75.2 |
74.6 |
80.2 |
84 |
131 |
63.69 |
800
n-gram
774
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.20 |
12.93 |
14.3 |
16.1 |
16.2 |
0.99945 |
64 |
92.7 |
93.3 |
100.3 |
102 |
104 |
63.739 |
128 |
119.7 |
128 |
144 |
151 |
163 |
127.247 |
256 |
199 |
229 |
256 |
260 |
274 |
253.28 |
384 |
278.5 |
335 |
375.3 |
385 |
419 |
378.33 |
512 |
384 |
457.8 |
517 |
592 |
646 |
499.7 |
768 |
649 |
696 |
1050 |
1300 |
1600 |
735.6 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1027.4 |
160
163
176
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
14.69 |
14.27 |
14.78 |
15.2 |
28.3 |
0.99908 |
n-gram |
8 |
37.7 |
37.2 |
38.4 |
39.6 |
66 |
7.9794 |
n-gram |
16 |
41.5 |
45.7 |
49.6 |
50.4 |
86 |
15.942 |
n-gram |
32 |
55.7 |
57.55 |
64.08 |
65.97 |
126.2 |
31.815 |
n-gram |
48 |
66.8 |
70.7 |
74.1 |
75.9 |
146 |
47.679 |
n-gram |
64 |
82.2 |
82.7 |
90 |
94.5 |
201 |
63.41 |
none |
1 |
13.68 |
13.337 |
13.54 |
14.4 |
21.9 |
0.999303 |
none |
8 |
37.0 |
36.5 |
37.9 |
38.5 |
60.4 |
7.9817 |
none |
16 |
41.4 |
45.4 |
46.6 |
48.0 |
77 |
15.943 |
none |
32 |
53.7 |
54.6 |
63 |
62.9 |
104.3 |
31.835 |
none |
48 |
64 |
68.1 |
70.4 |
74.6 |
134 |
47.687 |
none |
64 |
78.23 |
79.0 |
88.2 |
90.0 |
172.2 |
63.45 |
800
679
745
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
16.8 |
16.21 |
17.6 |
23.4 |
26.8 |
0.99923 |
n-gram |
64 |
109 |
107 |
122 |
167 |
261 |
63.45 |
n-gram |
128 |
130 |
121 |
159.0 |
218 |
382 |
126.27 |
n-gram |
256 |
236 |
242 |
288 |
461 |
780 |
249.28 |
n-gram |
384 |
342 |
360 |
414 |
659 |
1460 |
369.4 |
n-gram |
512 |
485 |
486 |
580 |
988.2 |
2349 |
485.5 |
none |
1 |
13.9 |
13.38 |
14.03 |
16.6 |
16.83 |
0.999467 |
none |
64 |
92 |
92 |
107.7 |
125.5 |
219 |
63.533 |
none |
128 |
119 |
107 |
146.7 |
206.4 |
372 |
126.35 |
none |
256 |
193.3 |
184 |
252.4 |
391 |
703 |
249.85 |
none |
384 |
301 |
326 |
367 |
579 |
1323 |
370.34 |
none |
512 |
409 |
436 |
478 |
773 |
2180 |
487.9 |
Language model |
# of streams |
Throughput (RTFX) |
---|---|---|
n-gram |
32 |
1700 |
none |
32 |
1660 |
160
n-gram
159
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.05 |
13.82 |
14.53 |
14.57 |
20.1 |
0.999527 |
8 |
37.9 |
37.3 |
38.7 |
39.7 |
59.8 |
7.9887 |
16 |
41.89 |
46.33 |
48.48 |
49.5 |
83 |
15.9757 |
32 |
55.6 |
57.08 |
64.50 |
65.9 |
127.5 |
31.9363 |
48 |
69.7 |
69.8 |
75 |
81.1 |
230.0 |
47.8753 |
64 |
86.6 |
84.1 |
95.6 |
101.4 |
298 |
63.795 |
800
n-gram
658
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
15.5 |
14.84 |
19.9 |
22.2 |
22.4 |
0.999523 |
64 |
100 |
92 |
115 |
189 |
299 |
63.792 |
128 |
133 |
120.0 |
148 |
290 |
521 |
127.353 |
256 |
253 |
234 |
309 |
661 |
1360 |
252.76 |
384 |
442.7 |
362 |
947.9 |
1644 |
2427 |
369.62 |
512 |
718 |
491 |
1940 |
2780 |
3940 |
480.6 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
890 |
160
n-gram
153
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.43 |
14.28 |
14.81 |
15.7 |
18.8 |
0.999447 |
8 |
38.1 |
38.2 |
39.5 |
40.0 |
42.6 |
7.9873 |
16 |
44.0 |
48.7 |
50.4 |
51.0 |
55.1 |
15.9677 |
32 |
58.7 |
62.28 |
69.3 |
70.73 |
74.3 |
31.9153 |
48 |
74.44 |
79.35 |
83.2 |
84.7 |
91.6 |
47.843 |
64 |
91.10 |
95.3 |
102.4 |
105.8 |
114 |
63.741 |
800
n-gram
685
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
15.2 |
15.12 |
16.61 |
16.83 |
18.5 |
0.999447 |
64 |
101 |
103 |
124.1 |
129 |
141.2 |
63.72 |
128 |
152 |
150.9 |
196 |
213.1 |
280 |
126.90 |
256 |
271 |
280 |
404 |
483 |
502 |
250.9 |
384 |
387 |
399.2 |
575 |
796 |
860 |
372.0 |
512 |
588 |
567 |
1110 |
1410 |
1497 |
490.2 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1870 |
160
n-gram
129
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
16.71 |
16.26 |
17.58 |
18.32 |
52.3 |
0.99827 |
8 |
43.8 |
42.5 |
46.3 |
47.8 |
120.0 |
7.9649 |
16 |
51.1 |
54.1 |
57.42 |
59.04 |
144 |
15.925 |
32 |
66.7 |
68.4 |
75.2 |
77.8 |
183 |
31.768 |
48 |
86 |
87.4 |
96.1 |
98.6 |
295.0 |
47.49 |
64 |
106.2 |
104.5 |
115.4 |
128 |
326 |
63.28 |
800
n-gram
425
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
24.4 |
23.65 |
26.3 |
46.8 |
51.2 |
0.99843 |
64 |
146 |
137 |
184.5 |
254 |
359 |
63.33 |
128 |
208 |
187 |
294 |
444 |
612 |
124.9 |
256 |
367 |
311 |
631 |
1080 |
1333 |
245.6 |
384 |
511 |
433 |
1109 |
1430 |
1879.2 |
362.1 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1342 |
160
none
83
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
15.23 |
14.94 |
15.7 |
16.5 |
20.74 |
0.99865 |
8 |
41.3 |
41.0 |
43.0 |
45.9 |
58.9 |
7.9627 |
16 |
48.0 |
50.6 |
54.7 |
58 |
84.4 |
15.887 |
32 |
66.2 |
65.0 |
75.4 |
80.3 |
146 |
31.624 |
48 |
93.9 |
96.4 |
107.1 |
110.4 |
223 |
47.238 |
64 |
117 |
119 |
127.8 |
132.4 |
290 |
62.69 |
800
none
346
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
18.40 |
17.73 |
20.7 |
20.9 |
22.3 |
0.99858 |
64 |
130 |
126 |
179 |
254 |
320 |
62.55 |
128 |
220 |
204 |
304 |
459 |
583 |
122.97 |
256 |
418.6 |
368 |
653 |
1097 |
1416 |
236.20 |
none
# of streams |
Throughput (RTFX) |
---|---|
32 |
947.3 |
160
n-gram
176
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.90 |
13.84 |
14.40 |
15.2 |
17.7 |
0.999357 |
8 |
37.5 |
37.4 |
38.5 |
39.1 |
42.1 |
7.98650 |
16 |
37.2 |
33.5 |
47.31 |
48.26 |
50.2 |
15.970 |
32 |
53.97 |
56.5 |
62.9 |
64.62 |
72 |
31.896 |
48 |
66.9 |
69.34 |
72.9 |
75.5 |
105.8 |
47.794 |
64 |
79.6 |
80.5 |
85.6 |
93 |
130 |
63.675 |
800
n-gram
787
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
15.1 |
14.99 |
16.5 |
16.30 |
20.7 |
0.999367 |
64 |
96 |
98 |
112 |
115.3 |
122 |
63.72 |
128 |
115.8 |
113.3 |
148.1 |
159.4 |
182.66 |
127.21 |
256 |
182.3 |
185 |
259.2 |
264.5 |
288 |
253.07 |
384 |
274 |
331 |
378 |
389 |
430.0 |
377.71 |
512 |
379 |
450 |
517 |
592 |
670 |
499.2 |
768 |
608 |
683 |
900.6 |
1060 |
1250 |
734.9 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
2010 |
160
n-gram
160
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.960 |
13.48 |
14.09 |
19.76 |
22.19 |
0.999 |
8 |
38.4 |
37.4 |
38.8 |
42.2 |
69.2 |
7.97650 |
16 |
41.7 |
43 |
47.9 |
52.8 |
102.6 |
15.9313 |
32 |
54.9 |
52.3 |
62.9 |
67.1 |
158 |
31.789 |
48 |
70.4 |
67.8 |
74.3 |
96 |
252.1 |
47.576 |
64 |
85.51 |
81.0 |
91.3 |
122 |
323 |
63.271 |
800
n-gram
478
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
16.33 |
14.92 |
21.4 |
21.4 |
21.19 |
0.999270 |
64 |
111.9 |
90 |
221 |
269 |
308 |
63.309 |
128 |
166 |
129.3 |
385 |
481 |
561 |
125.58 |
256 |
319 |
231.4 |
735 |
1124 |
1293 |
246.93 |
384 |
560 |
356.1 |
1426 |
1979 |
2240 |
363.93 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1023 |
160
n-gram
150
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.44 |
14.34 |
14.89 |
15.97 |
16.9 |
0.998960 |
8 |
29.9 |
29.5 |
31.0 |
33.8 |
43.6 |
7.978 |
16 |
44.53 |
49.3 |
51.0 |
52.0 |
54.0 |
15.9373 |
32 |
58.3 |
61.7 |
69.3 |
70.3 |
75 |
31.838 |
48 |
73.9 |
77.8 |
82 |
84 |
83.8 |
47.7037 |
64 |
88.6 |
92.2 |
99 |
102 |
110 |
63.49 |
800
n-gram
664
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
15.6 |
15.4 |
16.7 |
16.19 |
16.19 |
0.998777 |
64 |
97.4 |
112 |
122.3 |
126.3 |
129.5 |
63.42 |
128 |
137 |
139 |
160 |
163.6 |
173 |
126.33 |
256 |
224 |
254 |
294 |
304 |
317.2 |
250.16 |
384 |
324 |
380 |
441 |
460 |
477 |
370.8 |
512 |
432 |
515 |
574 |
584 |
597 |
490.2 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1560 |
100
n-gram
302
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
7.21 |
6.84 |
8.05 |
9.04 |
13.67 |
0.99944 |
8 |
9.665 |
9.03 |
10.77 |
12.7 |
26.4 |
7.9904 |
16 |
12.55 |
11.76 |
13.69 |
15.1 |
40.6 |
15.9693 |
32 |
19.52 |
18.79 |
22.77 |
24.6 |
74.9 |
31.891 |
48 |
23.21 |
21.4 |
27.7 |
31.2 |
105 |
47.772 |
64 |
27.72 |
25.10 |
33.4 |
37.9 |
147 |
63.617 |
800
n-gram
1214
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.3 |
11.90 |
15.7 |
16.0 |
15.87 |
0.999513 |
64 |
50.0 |
43.5 |
80 |
141 |
188 |
63.593 |
128 |
81 |
60.4 |
154 |
265 |
350 |
126.55 |
256 |
129.5 |
92 |
276 |
478 |
636 |
250.74 |
384 |
177 |
120.8 |
375 |
680 |
1010 |
372.6 |
512 |
222.9 |
144.0 |
525 |
968 |
1300 |
492.1 |
768 |
354 |
209 |
1058 |
1462 |
2118 |
724.3 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
2184 |
100
n-gram
185
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.22 |
12.90 |
14.09 |
15.1 |
19.97 |
0.999373 |
8 |
21.91 |
21.99 |
23.93 |
25.15 |
40.0 |
7.9874 |
16 |
19.34 |
16.92 |
28.7 |
31.0 |
45.6 |
15.963 |
32 |
35.8 |
35.7 |
41.3 |
43.4 |
79.5 |
31.877 |
48 |
43.6 |
42.9 |
49.5 |
53.4 |
97.6 |
47.761 |
64 |
47.2 |
45.4 |
52.2 |
56.1 |
150.2 |
63.613 |
800
n-gram
943
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
17.3 |
16.7 |
21.2 |
22.13 |
23.0 |
0.999397 |
64 |
70.9 |
63.4 |
108 |
166 |
213 |
63.53 |
128 |
116.5 |
99.9 |
186 |
288 |
367 |
126.47 |
256 |
178 |
146 |
312 |
517 |
681 |
250.24 |
384 |
249.8 |
200.3 |
456 |
756 |
1169 |
371.7 |
512 |
338 |
256 |
693 |
1210 |
1556 |
489.9 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1900 |
160
n-gram
146
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.74 |
13.47 |
13.74 |
13.86 |
23.71 |
0.999230 |
8 |
29.4 |
28.8 |
31.4 |
32.6 |
56.1 |
7.9813 |
16 |
44.2 |
43.3 |
45.9 |
46.4 |
83.1 |
15.9387 |
32 |
58.7 |
56.9 |
67.0 |
70.5 |
123 |
31.810 |
48 |
65.8 |
64.0 |
70.1 |
72.1 |
151.0 |
47.651 |
64 |
83 |
80.0 |
88.8 |
89.7 |
241 |
63.37 |
800
n-gram
594
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.64 |
14.21 |
14.56 |
19.71 |
20.20 |
0.999353 |
64 |
101 |
95 |
114 |
152.3 |
258 |
63.426 |
128 |
152 |
148.5 |
173 |
243 |
423 |
126.16 |
256 |
272 |
273 |
308 |
465 |
886 |
248.68 |
384 |
393 |
388 |
447 |
703 |
1568 |
368.13 |
512 |
569 |
512.9 |
661 |
1322 |
3840 |
484.0 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
3142 |
160
n-gram
165
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.24 |
13.26 |
13.65 |
13.77 |
15.42 |
1.000 |
8 |
26.2 |
26.71 |
28.17 |
28.5 |
29.9 |
7.9917 |
16 |
39.9 |
39.3 |
44.0 |
44 |
46 |
15.9757 |
32 |
51.7 |
52.2 |
58.34 |
59.9 |
62.9 |
31.9 |
48 |
56.2 |
57.45 |
61.4 |
62.6 |
64.3 |
47.8980 |
64 |
70.10 |
71.5 |
75.8 |
77.2 |
81.2 |
63.820 |
800
n-gram
672
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.91 |
14.04 |
14.9 |
15.18 |
15.38 |
1.000 |
64 |
96 |
94 |
112 |
115 |
121 |
63.77 |
128 |
134 |
143 |
158 |
167 |
178 |
127.300 |
256 |
238 |
268 |
298 |
306 |
317 |
253.35 |
384 |
350 |
393.8 |
440 |
449 |
466 |
378.2 |
512 |
455 |
513 |
599 |
643 |
689 |
499.4 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1256 |
160
n-gram
135
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.483 |
13.485 |
13.75 |
13.84 |
14.87 |
0.999457 |
8 |
32.5 |
32.15 |
34.47 |
39.4 |
44.6 |
7.98993 |
16 |
35.2 |
31.9 |
46.0 |
48.558 |
50.6 |
15.974 |
32 |
54.7 |
56.0 |
64.4 |
66.1 |
68.6 |
31.919 |
48 |
68.5 |
78 |
85.1 |
86.0 |
88.0 |
47.846 |
64 |
82.6 |
93 |
102.0 |
103.6 |
109.0 |
63.759 |
800
n-gram
624
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.92 |
13.94 |
14.6 |
15.0 |
14.97 |
0.999463 |
64 |
91 |
93 |
111.8 |
139 |
157.1 |
63.72 |
128 |
156.8 |
160.4 |
217 |
228 |
291 |
126.77 |
256 |
278 |
292.6 |
389 |
498 |
523 |
251.01 |
384 |
393 |
413 |
640 |
789.7 |
840 |
372.43 |
512 |
549 |
563 |
940 |
1126 |
1510 |
490.28 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1043.2 |
160
n-gram
146
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.96 |
14.690 |
15.64 |
16.17 |
35.8 |
0.998813 |
8 |
23.76 |
22.26 |
26.2 |
31.3 |
60.2 |
7.980 |
16 |
42.63 |
41.39 |
44.3 |
45.60 |
112 |
15.934 |
32 |
56.3 |
55.0 |
62.8 |
65.0 |
147.9 |
31.822 |
48 |
63.3 |
60.7 |
66.9 |
71 |
153.3 |
47.60 |
64 |
76.1 |
74.3 |
81.7 |
84.7 |
149 |
63.48 |
800
n-gram
517
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
18.88 |
18.33 |
20.75 |
31.7 |
34.8 |
0.99894 |
64 |
123 |
113 |
175 |
229.2 |
258 |
63.43 |
128 |
193 |
174 |
241.4 |
428 |
508 |
125.6 |
256 |
288 |
282 |
410 |
737 |
840 |
249.0 |
384 |
415 |
398 |
560 |
1200 |
1280 |
368.5 |
512 |
609 |
531 |
1160 |
1640 |
1980 |
483.9 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1000 |
160
n-gram
109
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
17.15 |
15.99 |
20.91 |
31.00 |
39.3 |
0.999410 |
8 |
42.2 |
39.1 |
52.2 |
63 |
74.7 |
7.9868 |
16 |
58.2 |
56.2 |
69.5 |
79.0 |
102.8 |
15.9650 |
32 |
71.0 |
70.8 |
83.7 |
96.9 |
120.3 |
31.906 |
48 |
91.1 |
92.2 |
109 |
117 |
153 |
47.81 |
64 |
114.0 |
115 |
137.4 |
155 |
181.3 |
63.686 |
800
n-gram
457
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
23.82 |
21.18 |
36.8 |
37.6 |
38.6 |
0.999410 |
64 |
137 |
135.6 |
192 |
201 |
205 |
63.67 |
128 |
219 |
227 |
284 |
298 |
322 |
126.79 |
256 |
400 |
404 |
525 |
545 |
756 |
251.1 |
384 |
603 |
595 |
937 |
1060 |
1216 |
370.1 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
762.4 |
160
n-gram
104
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
15.54 |
15.53 |
15.91 |
16.09 |
16.9 |
0.999350 |
8 |
40.6 |
42.1 |
45.3 |
45.8 |
46.8 |
7.9871 |
16 |
43.3 |
42.2 |
44.4 |
58.7 |
62.9 |
15.965 |
32 |
64.5 |
64.5 |
66.4 |
67.9 |
88.1 |
31.887 |
48 |
88.7 |
87.9 |
92.4 |
94.7 |
140 |
47.749 |
64 |
111.3 |
109.4 |
119 |
123 |
184 |
63.555 |
800
n-gram
464
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
16.29 |
16.32 |
16.84 |
16.93 |
16.97 |
0.999337 |
64 |
117.1 |
112.6 |
143 |
155 |
156 |
63.67 |
128 |
177 |
193 |
209 |
213 |
225 |
126.9 |
256 |
309.2 |
352 |
392 |
405 |
430 |
251.86 |
384 |
464 |
539 |
643 |
764 |
840 |
372.82 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
861 |
160
95
100
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
20.32 |
19.88 |
20.28 |
20.6 |
39.8 |
0.99874000000000007 |
n-gram |
8 |
42.2 |
41.3 |
43.2 |
44.2 |
93 |
7.9712 |
n-gram |
16 |
51.5 |
54.4 |
62.66 |
64.6 |
105.8 |
15.936 |
n-gram |
32 |
71.3 |
75.0 |
80 |
84.2 |
161 |
31.785 |
n-gram |
48 |
103.9 |
105.5 |
113.1 |
116.4 |
258 |
47.560 |
n-gram |
64 |
126.8 |
129.0 |
140.4 |
145.9 |
351 |
63.21 |
none |
1 |
18.93 |
18.52 |
18.67 |
18.77 |
32.36 |
0.998977 |
none |
8 |
46.7 |
41.0 |
56.8 |
57.5 |
89.5 |
7.9727 |
none |
16 |
49.0 |
51.5 |
60.0 |
63 |
106 |
15.930 |
none |
32 |
69.8 |
73.67 |
75.9 |
80 |
148 |
31.799 |
none |
48 |
92 |
83.5 |
108 |
110 |
244 |
47.561 |
none |
64 |
116 |
125.6 |
138.4 |
143 |
341 |
63.238 |
800
413
435
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
21.76 |
21.41 |
21.97 |
28.94 |
29.54 |
0.999 |
n-gram |
64 |
134 |
139 |
173 |
206 |
294 |
63.38 |
n-gram |
128 |
216 |
201.2 |
254 |
331 |
512 |
125.8 |
n-gram |
256 |
397 |
419 |
474 |
641 |
1378 |
247.60 |
n-gram |
384 |
672 |
632 |
729 |
1514 |
4660 |
363.65 |
none |
1 |
18.92 |
18.53 |
18.9 |
23.07 |
23.18 |
0.999253 |
none |
64 |
127 |
129.3 |
158.2 |
189 |
262 |
63.423 |
none |
128 |
207.2 |
210 |
250 |
318 |
503 |
125.82 |
none |
256 |
355 |
368 |
448 |
620.7 |
1367 |
247.51 |
none |
384 |
594 |
588 |
663 |
1439 |
3949 |
364.1 |
Language model |
# of streams |
Throughput (RTFX) |
---|---|---|
n-gram |
32 |
1120 |
none |
32 |
1152 |
160
n-gram
92
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.517 |
19.30 |
19.74 |
19.85 |
28.77 |
0.999337 |
8 |
51.7 |
52.8 |
57.1 |
57.7 |
89.0 |
7.98800 |
16 |
51.0 |
52.7 |
62.6 |
66.7 |
106.3 |
15.9687 |
32 |
77.0 |
82 |
90.25 |
91.9 |
254.4 |
31.9007 |
48 |
110.4 |
104.9 |
121.2 |
165.9 |
472 |
47.815 |
64 |
212.7 |
138.8 |
485 |
850 |
1409 |
63.567 |
800
n-gram
396
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.50 |
19.91 |
20.982 |
29.4 |
29.75 |
0.999333 |
64 |
136 |
134 |
146 |
241 |
401 |
63.690 |
128 |
234 |
226 |
251.0 |
439 |
760 |
126.927 |
256 |
512 |
409 |
1137 |
1820 |
2619 |
245.97 |
384 |
1175 |
636.3 |
3370 |
5640 |
7580 |
355.6 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
561 |
160
n-gram
87
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.14 |
20.16 |
20.54 |
20.65 |
22.27 |
0.999210 |
8 |
44.7 |
45.2 |
46.5 |
46.9 |
51.0 |
7.9833 |
16 |
52.1 |
55.6 |
62.6 |
63.6 |
66.2 |
15.9600 |
32 |
74.9 |
80.8 |
87 |
88.7 |
91.8 |
31.8973 |
48 |
99 |
109 |
118.5 |
122.6 |
132 |
47.7760 |
64 |
125.0 |
135.5 |
149.5 |
152.9 |
186.4 |
63.543 |
800
n-gram
403
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.74 |
20.85 |
22.0 |
23.0 |
23.6 |
0.999233 |
64 |
149.3 |
147.8 |
191 |
205 |
246 |
63.49 |
128 |
261 |
257 |
388 |
408 |
457 |
126.03 |
256 |
448 |
458 |
618 |
832 |
997 |
247.94 |
384 |
754 |
697 |
1328 |
1373 |
1611 |
365.26 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1210 |
160
n-gram
82
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
22.15 |
21.774 |
22.85 |
23.38 |
54.8 |
0.99821 |
8 |
47.8 |
46.31 |
49.9 |
50.8 |
125.4 |
7.9637 |
16 |
56.8 |
59.7 |
66.8 |
68.5 |
133 |
15.9267 |
32 |
80.0 |
82.3 |
86.4 |
89.0 |
221.6 |
31.731 |
48 |
122 |
120 |
133 |
134.6 |
312 |
47.37 |
64 |
146.2 |
147 |
173.1 |
194 |
413 |
63.07 |
800
n-gram
315
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
28.47 |
27.98 |
30.22 |
46.4 |
49 |
0.998477 |
64 |
198 |
182.9 |
258 |
357 |
443 |
63.08 |
128 |
321 |
298 |
435 |
700 |
810 |
124.3 |
256 |
565 |
499 |
837 |
1380 |
1463 |
244.8 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1140 |
160
none
63
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.88 |
20.55 |
21.2 |
21.8 |
29.31 |
0.99801 |
8 |
60.3 |
60.8 |
68.1 |
70.1 |
96.0 |
7.9430 |
16 |
62.4 |
64.2 |
77.0 |
80.5 |
102.8 |
15.861 |
32 |
97 |
103 |
117 |
123 |
208 |
31.507 |
48 |
135 |
138 |
159 |
169.8 |
336 |
46.92 |
800
none
233
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
23.7 |
22.86 |
28.20 |
28.35 |
28.60 |
0.998067 |
64 |
171 |
163 |
228.2 |
325 |
410 |
62.24 |
128 |
322 |
295 |
428 |
631 |
800 |
121.4 |
none
# of streams |
Throughput (RTFX) |
---|---|
32 |
659 |
160
n-gram
101
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.11 |
19.10 |
19.70 |
19.99 |
22.5 |
0.999147 |
8 |
54.2 |
54.3 |
57.2 |
57.6 |
59.7 |
7.9822 |
16 |
51.9 |
55.3 |
62.8 |
64.7 |
73 |
15.9650 |
32 |
70.7 |
75.3 |
85.1 |
86.9 |
110 |
31.858 |
48 |
97 |
102.8 |
113.9 |
116.4 |
170 |
47.689 |
64 |
117 |
122.1 |
137.4 |
140.1 |
231 |
63.469 |
800
n-gram
466
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.90 |
20.10 |
20.8 |
21.5 |
22.1 |
0.999163 |
64 |
123 |
130.8 |
142.3 |
145.6 |
148.1 |
63.652 |
128 |
207 |
226 |
244.0 |
248 |
257 |
126.77 |
256 |
335 |
377 |
444 |
489 |
533 |
251.10 |
384 |
511 |
592 |
675 |
887 |
1017 |
371.0 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1300 |
160
n-gram
92
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.49 |
18.81 |
19.30 |
28.0 |
33.1 |
0.998877 |
8 |
43.9 |
42.2 |
58.6 |
63.0 |
107 |
7.9639 |
16 |
51.11 |
51.6 |
61.9 |
66.1 |
128 |
15.910 |
32 |
74 |
61.2 |
82 |
107 |
261 |
31.718 |
48 |
110.8 |
104.4 |
121 |
200 |
469 |
47.385 |
64 |
163 |
128 |
206.5 |
485 |
937 |
62.930 |
800
n-gram
303
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
21.39 |
19.44 |
28.58 |
28.74 |
29.0 |
0.999030 |
64 |
156 |
130.5 |
309 |
374 |
429 |
63.106 |
128 |
270 |
223 |
559.6 |
689.4 |
796 |
124.60 |
256 |
647 |
428 |
1526 |
2135 |
2371 |
243.00 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
675 |
160
n-gram
87
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.13 |
20.12 |
20.57 |
21.0 |
22.10 |
0.998493 |
8 |
43.4 |
43.5 |
44.9 |
45.7 |
56.4 |
7.969 |
16 |
51.0 |
54.6 |
61.3 |
63.5 |
65.4 |
15.927 |
32 |
75.8 |
80.6 |
85.0 |
88 |
93 |
31.7973 |
48 |
108.1 |
114.1 |
127 |
131 |
146.2 |
47.557 |
64 |
132 |
140.2 |
157.8 |
168 |
199 |
63.27 |
800
n-gram
402
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.85 |
20.81 |
21.90 |
22.3 |
22.3 |
0.998337 |
64 |
137 |
142.0 |
161.46 |
166.0 |
170.9 |
63.279 |
128 |
234 |
242 |
282 |
288 |
285.7 |
125.34 |
256 |
405 |
455 |
491 |
498 |
514 |
246.6 |
384 |
594 |
670 |
815 |
838 |
1070 |
361.07 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1010 |
100
n-gram
234
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
9.53 |
9.21 |
10.21 |
10.9 |
17.8 |
0.999367 |
8 |
16.36 |
15.7 |
17.95 |
19.19 |
39.9 |
7.9861 |
16 |
21.6 |
20.4 |
23.18 |
24.78 |
64.3 |
15.9567 |
32 |
26.2 |
24.4 |
30.7 |
34.1 |
112.7 |
31.848 |
48 |
32.70 |
28.15 |
40.5 |
45.7 |
170 |
47.693 |
64 |
38.4 |
33.1 |
47.6 |
53.7 |
203 |
63.524 |
800
n-gram
1039
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.71 |
14.23 |
19.1 |
19.5 |
20.0 |
0.999383 |
64 |
69.0 |
57 |
122.2 |
198.0 |
257.0 |
63.439 |
128 |
112.6 |
91.2 |
209 |
340.2 |
438.7 |
126.12 |
256 |
156.5 |
116 |
314 |
545 |
733 |
250.06 |
384 |
225 |
162 |
455 |
787 |
1181 |
371.3 |
512 |
294 |
206.7 |
648 |
1145 |
1522 |
490.2 |
768 |
506 |
299.8 |
1408 |
2066 |
2860 |
716.19 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
2310 |
100
n-gram
111
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
18.57 |
18.31 |
19.26 |
20.18 |
26.3 |
0.99915 |
8 |
27.1 |
23.39 |
38.9 |
40.0 |
48.2 |
7.9818 |
16 |
34.6 |
31.1 |
45.2 |
47.2 |
67.8 |
15.9463 |
32 |
47.6 |
46.8 |
53.2 |
55.3 |
89.6 |
31.869 |
48 |
61.0 |
58.9 |
67 |
70 |
168 |
47.716 |
64 |
77.1 |
72.8 |
81.6 |
85.6 |
292 |
63.528 |
800
n-gram
640
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
22.74 |
22.06 |
27.020 |
29.78 |
30.02 |
0.999163 |
64 |
104.8 |
101.2 |
154.3 |
220 |
277 |
63.436 |
128 |
158 |
136.3 |
225.9 |
341 |
460 |
126.17 |
256 |
267 |
236 |
428.0 |
664.1 |
1067 |
249.01 |
384 |
399 |
332 |
710 |
1251 |
1551 |
368.7 |
512 |
630 |
431.8 |
1500 |
2220 |
3090 |
482.7 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1950 |
160
n-gram
123
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
10.405 |
10.279 |
10.517 |
10.687 |
14.8 |
0.99947 |
8 |
20.22 |
19.04 |
25.5 |
25.41 |
35.9 |
7.9865 |
16 |
29.8 |
28.9 |
33.0 |
34.5 |
58 |
15.9577 |
32 |
49.1 |
47.7 |
51.4 |
52.9 |
112 |
31.831 |
48 |
67.6 |
65.7 |
70 |
72.3 |
169 |
47.631 |
64 |
84.7 |
81.5 |
90.4 |
93.0 |
262 |
63.360 |
800
n-gram
541
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.93 |
12.53 |
13.04 |
17.0 |
17.43 |
0.999427 |
64 |
88.5 |
85.0 |
99.2 |
133.0 |
224 |
63.506 |
128 |
162.6 |
161.9 |
185 |
260 |
445 |
126.01 |
256 |
316 |
316 |
348 |
510 |
1148 |
248.27 |
384 |
486 |
468 |
518 |
757 |
2120 |
367.4 |
512 |
710 |
613 |
740 |
1550 |
4790 |
481.2 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
2719 |
160
n-gram
139
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
10.95 |
10.98 |
11.38 |
11.48 |
13.1 |
1.000 |
8 |
18.60 |
19.04 |
20.51 |
20.85 |
22.0 |
7.99413 |
16 |
29.7 |
29.8 |
32.4 |
33.1 |
33.5 |
15.9817 |
32 |
44.6 |
45.2 |
48.4 |
49.3 |
53.0 |
31.944 |
48 |
59.0 |
60.6 |
64.1 |
64.9 |
67.5 |
47.891 |
64 |
75.5 |
77.4 |
85 |
85 |
87.1 |
63.802 |
800
n-gram
642
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
11.87 |
11.94 |
12.8 |
13.3 |
13.29 |
0.999583 |
64 |
76.3 |
77.5 |
82.9 |
86 |
90 |
63.814 |
128 |
138.2 |
148.6 |
162.2 |
168 |
176.3 |
127.297 |
256 |
254 |
282 |
304.0 |
311 |
325 |
253.28 |
384 |
368 |
417 |
449 |
458 |
501 |
377.96 |
512 |
496 |
551 |
630 |
722 |
860 |
499.0 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1169 |
160
n-gram
105
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
11.11 |
11.11 |
11.37 |
11.46 |
12.52 |
1.000 |
8 |
23.65 |
24.17 |
28.2 |
30.8 |
32.7 |
7.9907 |
16 |
40.2 |
39.9 |
44 |
42.2 |
45.7 |
15.9683 |
32 |
62 |
66.8 |
76 |
78 |
81 |
31.909 |
48 |
85.6 |
95.9 |
102.2 |
103.9 |
108 |
47.831 |
64 |
110.0 |
121.5 |
130 |
132 |
146 |
63.63 |
800
n-gram
457
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
10.9 |
10.9 |
11.5 |
12.0 |
11.83 |
0.999580 |
64 |
106.5 |
117 |
130.1 |
140 |
159.0 |
63.662 |
128 |
210.0 |
215.5 |
316 |
331 |
377 |
126.370 |
256 |
373 |
399.0 |
517.1 |
680 |
704 |
249.13 |
384 |
578 |
586 |
878 |
1103 |
1420 |
368.3 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
949 |
160
n-gram
122
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
11.34 |
10.99 |
12.13 |
12.76 |
31.54 |
0.99891 |
8 |
19.95 |
19.11 |
21.61 |
22.78 |
60.0 |
7.975 |
16 |
32.1 |
30.6 |
35.1 |
36.9 |
101.4 |
15.942 |
32 |
51.5 |
49.4 |
58.4 |
61.1 |
161 |
31.80 |
48 |
68.3 |
66.0 |
73 |
77 |
221 |
47.57 |
64 |
82.8 |
80.2 |
87.7 |
93 |
225.3 |
63.459 |
800
n-gram
471
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
17.46 |
16.90 |
21.3 |
30.9 |
32.7 |
0.99894 |
64 |
111.6 |
108 |
128.8 |
165.6 |
256 |
63.46 |
128 |
211 |
188 |
304 |
398 |
564 |
125.2 |
256 |
362 |
332 |
539 |
924 |
999 |
247.7 |
384 |
535 |
472 |
834 |
1330 |
1640 |
366.9 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
890 |
160
n-gram
91
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.98 |
12.60 |
17.97 |
29.14 |
37.2 |
0.999527 |
8 |
35.6 |
33.8 |
43.9 |
58.1 |
78 |
7.9886 |
16 |
46.8 |
44.7 |
54.8 |
66.6 |
90 |
15.967 |
32 |
76 |
78 |
94 |
110.3 |
133 |
31.894 |
48 |
102 |
104.3 |
122 |
140 |
173 |
47.772 |
64 |
132.6 |
134 |
154.5 |
180 |
220 |
63.60 |
800
n-gram
390
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
21.5 |
18.3 |
36.4 |
37.6 |
38.0 |
0.999510 |
64 |
161 |
157 |
209 |
218 |
224 |
63.645 |
128 |
288 |
288 |
366 |
369 |
469 |
126.52 |
256 |
505 |
493 |
740 |
800 |
1010 |
248.3 |
384 |
918 |
771 |
1649 |
1870 |
2122 |
365.8 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
730 |
160
n-gram
76
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
11.44 |
11.44 |
11.78 |
11.93 |
12.74 |
0.999493 |
8 |
26.3 |
26.1 |
25.9 |
26.6 |
30.4 |
7.9909 |
16 |
44.2 |
44.1 |
45.6 |
46.0 |
48.1 |
15.9690 |
32 |
73.76 |
73.48 |
74.99 |
75.71 |
127.6 |
31.863 |
48 |
109.9 |
109.1 |
114.1 |
119 |
183 |
47.670 |
64 |
135 |
138 |
154 |
158.2 |
286 |
63.394 |
800
n-gram
376
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.47 |
13.60 |
13.90 |
14.10 |
14.19 |
0.999437 |
64 |
132 |
141 |
150 |
153 |
157 |
63.626 |
128 |
250 |
268 |
285.1 |
289.3 |
299 |
126.57 |
256 |
428.0 |
491 |
549 |
642 |
693 |
249.61 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
822.7 |
160
78
80
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
13.49 |
13.23 |
13.73 |
13.98 |
25.03 |
0.999173 |
n-gram |
8 |
33.8 |
33.25 |
35.3 |
36.0 |
61.0 |
7.9818 |
n-gram |
16 |
40.9 |
43.8 |
46.5 |
47.2 |
94.9 |
15.9377 |
n-gram |
32 |
71.5 |
74.98 |
77.5 |
80.1 |
166 |
31.780 |
n-gram |
48 |
108 |
112.4 |
123 |
127 |
295 |
47.506 |
n-gram |
64 |
140 |
141.8 |
157 |
201 |
397 |
63.161 |
none |
1 |
12.11 |
11.91 |
12.08 |
12.23 |
18.34 |
0.999393 |
none |
8 |
29.6 |
31.0 |
31.9 |
32.2 |
50.9 |
7.98353 |
none |
16 |
38.8 |
37.4985 |
39.31 |
39.88 |
89.0 |
15.941 |
none |
32 |
72.01 |
75.62 |
76.83 |
77.44 |
168.7 |
31.776 |
none |
48 |
110.6 |
113.04 |
114.94 |
123 |
273 |
47.540 |
none |
64 |
144 |
143 |
149.6 |
187.5 |
386 |
63.168 |
800
366
356
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
15.33 |
15.01 |
15.80 |
21.7 |
21.9 |
0.999277 |
n-gram |
64 |
133 |
140.0 |
154 |
192 |
278 |
63.39 |
n-gram |
128 |
234 |
256 |
272.8 |
364 |
538 |
125.69 |
n-gram |
256 |
434 |
461 |
529 |
717 |
1519 |
247.05 |
none |
1 |
12.90 |
12.60 |
13.3 |
15.62 |
15.90 |
0.999467 |
none |
64 |
135.7 |
137 |
148.8 |
183 |
272 |
63.408 |
none |
128 |
212 |
187 |
266 |
349 |
528 |
125.743 |
none |
256 |
418 |
432 |
522 |
714 |
1520 |
246.93 |
Language model |
# of streams |
Throughput (RTFX) |
---|---|---|
n-gram |
32 |
992 |
none |
32 |
1006 |
160
n-gram
79
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.67 |
12.57 |
13.04 |
13.22 |
18.5 |
0.999553 |
8 |
28.9 |
28.2 |
32.5 |
33.2 |
48.339 |
7.99040 |
16 |
40.8 |
44.1 |
46.9 |
48.3 |
83.3 |
15.9733 |
32 |
66 |
60.2 |
79.43 |
84.9 |
152 |
31.9123 |
48 |
105.6 |
106.6 |
122.5 |
152 |
462 |
47.803 |
64 |
311 |
149.3 |
970 |
1390 |
2030 |
63.339 |
800
n-gram
348
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.16 |
13.77 |
14.98 |
20.9 |
21.0 |
0.999537 |
64 |
137.6 |
138.7 |
152 |
219 |
345 |
63.679 |
128 |
241 |
244 |
272.2 |
424 |
680 |
126.827 |
256 |
585 |
466 |
1300 |
1980 |
2752 |
245.79 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
576 |
160
n-gram
73
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.233 |
13.246 |
13.705 |
13.85 |
15.65 |
0.999467 |
8 |
28.4 |
26.6 |
32.4 |
32.9 |
35.1 |
7.9891 |
16 |
44.40 |
50.9 |
52.48 |
53.3 |
55.7 |
15.9660 |
32 |
72.5 |
88 |
91 |
92.7 |
96.8 |
31.895 |
48 |
122.9 |
132.6 |
140.4 |
144.3 |
157 |
47.713 |
64 |
174 |
178.6 |
199.4 |
214 |
287 |
63.45 |
800
n-gram
352
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.87 |
13.93 |
15.0 |
15.80 |
16.6 |
0.999463 |
64 |
154 |
155 |
193 |
205 |
241 |
63.494 |
128 |
272.4 |
272 |
412 |
447 |
524.1 |
125.73 |
256 |
522 |
516 |
920 |
1080 |
1395 |
246.61 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1138 |
160
n-gram
69
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
15.76 |
15.44 |
16.60 |
16.98 |
47.5 |
0.99842 |
8 |
34.5 |
33.5 |
37.2 |
38.4 |
73.3 |
7.9684 |
16 |
45.6 |
47.9 |
52.1 |
52.85 |
131 |
15.928 |
32 |
78 |
84 |
89.3 |
91.8 |
192.8 |
31.73 |
48 |
111.0 |
115.8 |
141 |
149 |
369 |
47.38 |
64 |
173 |
167 |
206 |
272 |
504 |
62.86 |
800
n-gram
278
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
22.49 |
21.90 |
25.89 |
41.3 |
42.4 |
0.99864 |
64 |
211 |
196 |
277 |
376 |
442 |
63.01 |
128 |
357 |
312 |
390 |
870 |
990 |
124.15 |
256 |
611 |
579 |
1096 |
1470 |
1860 |
243.1 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1010 |
160
none
56
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.11 |
13.97 |
14.43 |
14.94 |
18.7 |
0.99870 |
8 |
37.0 |
39.6 |
44.4 |
45.5 |
51.2 |
7.964 |
16 |
53.7 |
59.3 |
63 |
65 |
87 |
15.889 |
32 |
94 |
101.7 |
110 |
115 |
164 |
31.567 |
48 |
138 |
145 |
162.6 |
169.5 |
315 |
47.02 |
800
none
230
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
17.03 |
16.664 |
19.10 |
19.3 |
19.5 |
0.998643 |
64 |
164.8 |
156.1 |
215.6 |
281 |
368 |
62.35 |
128 |
384 |
354 |
539.3 |
635 |
776 |
120.7 |
none
# of streams |
Throughput (RTFX) |
---|---|
32 |
690 |
160
n-gram
83
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.72 |
12.78 |
13.37 |
13.64 |
15.70 |
0.999367 |
8 |
28.7 |
28.86 |
29.85 |
30.18 |
32.5 |
7.98897 |
16 |
37.7 |
41.7 |
44.0 |
44.6 |
47.7 |
15.9670 |
32 |
67.4 |
73.2 |
76.7 |
78.5 |
107 |
31.861 |
48 |
104 |
110.1 |
117.4 |
120.0 |
200 |
47.640 |
64 |
128.6 |
137.5 |
148.9 |
154.8 |
277 |
63.383 |
800
n-gram
380
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.89 |
14.207 |
14.90 |
15.45 |
15.469 |
0.999393 |
64 |
132.3 |
138.1 |
148.4 |
151.3 |
160.1 |
63.620 |
128 |
223.6 |
243 |
272.7 |
277.5 |
286 |
126.66 |
256 |
397 |
444 |
515 |
614 |
670 |
249.81 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1180 |
160
n-gram
77
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.83 |
12.54 |
13.11 |
13.80 |
19.73 |
0.999317 |
8 |
26.7 |
25.6 |
27.4 |
31.7 |
56 |
7.9811 |
16 |
40.1 |
42.6 |
44.3 |
50.1 |
96.7 |
15.933 |
32 |
72.2 |
73.14 |
79.8 |
104 |
245.0 |
31.7390 |
48 |
118.4 |
109.3 |
128 |
186 |
472.0 |
47.406 |
64 |
214 |
146.39 |
456 |
801 |
1300 |
62.971 |
800
n-gram
281
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.33 |
13.14 |
19.14 |
19.3 |
19.4 |
0.999327 |
64 |
152 |
136.6 |
271.7 |
324.5 |
367 |
63.226 |
128 |
277 |
251 |
527 |
634 |
719 |
124.95 |
256 |
710 |
495 |
1557 |
2217 |
2680 |
244.18 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
683 |
160
n-gram
72
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.45 |
13.35 |
13.96 |
14.8 |
19.0 |
0.998973 |
8 |
32.32 |
32.25 |
36.72 |
37.7 |
39.0 |
7.9764 |
16 |
45.5 |
51.14 |
55.3 |
56.0 |
57.4 |
15.932 |
32 |
80.8 |
89.73 |
94.8 |
96.6 |
100.1 |
31.777 |
48 |
123 |
131.6 |
139.6 |
141.8 |
147 |
47.506 |
64 |
171 |
178.3 |
201 |
202 |
243.6 |
63.09 |
800
n-gram
338
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.3 |
14.2 |
15.40 |
15.47 |
15.47 |
0.99879 |
64 |
156 |
162 |
174 |
174 |
178.0 |
63.151 |
128 |
257 |
267 |
302 |
313 |
310.46 |
125.09 |
256 |
471 |
520 |
573 |
584 |
607 |
245.3 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1020 |
100
n-gram
231
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
7.72 |
7.42 |
8.51 |
9.28 |
14.4 |
0.999450 |
8 |
10.01 |
9.49 |
10.88 |
11.72 |
29.1 |
7.9902 |
16 |
12.5 |
11.8 |
13.6 |
14.5 |
40.18 |
15.9707 |
32 |
18.7 |
18.0 |
21.7 |
23.80 |
69.3 |
31.896 |
48 |
24.6 |
23.4 |
27.8 |
30.73 |
100.0 |
47.782 |
64 |
30.1 |
27.9 |
34.3 |
37.5 |
145.1 |
63.623 |
800
n-gram
976
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.27 |
12.19 |
16.06 |
16.55 |
16.72 |
0.999483 |
64 |
59.6 |
53 |
88 |
130.9 |
170.1 |
63.646 |
128 |
94 |
76 |
149.0 |
248 |
330 |
126.587 |
256 |
161.6 |
134 |
290.4 |
491 |
653 |
250.59 |
384 |
234 |
190 |
426.2 |
709.8 |
1098.6 |
372.32 |
512 |
313 |
242 |
627 |
1127 |
1462 |
491.8 |
768 |
565 |
362 |
1492 |
2157 |
3080 |
718.8 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
2047 |
100
n-gram
75
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.24 |
13.04 |
13.94 |
14.92 |
20.29 |
0.999347 |
8 |
19.7 |
19.0 |
21.5 |
22.88 |
34.7 |
7.9879 |
16 |
34.5 |
33.8 |
36.8 |
39.2 |
58.8 |
15.9593 |
32 |
53.32 |
52.90 |
58.5 |
61.15 |
97 |
31.8627 |
48 |
76.8 |
73.5 |
80.3 |
84.1 |
285 |
47.700 |
64 |
180 |
116.8 |
201 |
538 |
1380 |
63.439 |
800
n-gram
475
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
16.99 |
16.44 |
21.38 |
23.25 |
23.9 |
0.99936 |
64 |
105.7 |
97.3 |
141.6 |
193.52 |
236.58 |
63.505 |
128 |
196 |
184 |
270 |
376 |
464 |
126.06 |
256 |
368.5 |
343 |
522.2 |
739 |
1241 |
248.56 |
384 |
632 |
502.9 |
1396 |
2090 |
2820 |
366.3 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1560 |
160
n-gram
140
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.26 |
13.05 |
13.30 |
13.44 |
21.9 |
0.999283 |
8 |
24.32 |
24.28 |
25.3 |
25.5 |
45.8 |
7.9827 |
16 |
32.9 |
32.0 |
33.4 |
33.9 |
61.9 |
15.948 |
32 |
50.5 |
48.9 |
53.1 |
54.2 |
113 |
31.816 |
48 |
65.0 |
62.8 |
67.9 |
70.8 |
162 |
47.6037 |
64 |
84.1 |
81.2 |
88.7 |
93.2 |
254 |
63.304 |
800
n-gram
579
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.91 |
13.45 |
14.0 |
19.21 |
19.55 |
0.99937000000000020 |
64 |
87.9 |
81.0 |
89 |
153.4 |
294.2 |
63.344 |
128 |
153 |
141.5 |
160 |
296 |
580 |
125.470 |
256 |
283.7 |
266.7 |
298 |
566 |
1312 |
246.22 |
384 |
407 |
375 |
433 |
915 |
2341 |
362.62 |
512 |
590 |
505 |
612 |
1511 |
4670 |
474.07 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
2693 |
160
n-gram
147
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
12.7 |
12.8 |
13.1 |
13.2 |
14.71 |
0.999537 |
8 |
20.10 |
19.97 |
20.99 |
21.48 |
27.6 |
7.9929 |
16 |
33.52 |
33.47 |
34.8 |
35.1 |
38.87 |
15.98 |
32 |
51.0 |
51.1 |
53.2 |
54.0 |
58.0 |
31.9370 |
48 |
61.8 |
62.3 |
67 |
69 |
73 |
47.885 |
64 |
75.4 |
77.2 |
84 |
85.9 |
91.1 |
63.8107 |
800
n-gram
677
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.28 |
13.4 |
14.20 |
14.80 |
14.87 |
0.999537 |
64 |
81.1 |
81.6 |
90.4 |
93.30 |
97.0 |
63.8047 |
128 |
139 |
142.3 |
155.4 |
160.5 |
173 |
127.313 |
256 |
248 |
267.7 |
292.4 |
299 |
318 |
253.42 |
384 |
325 |
374 |
416 |
427 |
448 |
378.39 |
512 |
449 |
503.0 |
601 |
656 |
780 |
500.0 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
984.3 |
160
n-gram
127
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.42 |
13.48 |
13.685 |
13.76 |
14.89 |
0.999457 |
8 |
32.0 |
31.9 |
37.4 |
37.1 |
38.5 |
7.9888 |
16 |
47.4 |
46.0 |
53.4 |
59.7 |
61.2 |
15.965 |
32 |
63.7 |
64.1 |
77.0 |
80.0 |
83.8 |
31.901 |
48 |
77.6 |
80 |
97.8 |
99.0 |
102.8 |
47.826 |
64 |
93.09 |
106 |
117.4 |
120.3 |
131.8 |
63.688 |
800
n-gram
569
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
13.86 |
13.88 |
14.6 |
15.04 |
15.43 |
0.999453 |
64 |
97.3 |
108 |
122 |
126 |
147.1 |
63.683 |
128 |
173 |
183 |
252 |
264 |
276 |
126.67 |
256 |
314 |
329 |
409.5 |
549 |
582 |
250.26 |
384 |
465 |
474 |
684 |
1000 |
1280 |
370.6 |
512 |
722 |
634.2 |
1440 |
1770 |
2330 |
487.9 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
691 |
160
n-gram
127
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.44 |
14.05 |
15.21 |
15.8 |
40.8 |
0.99850 |
8 |
27.0 |
26.1 |
27.9 |
28.7 |
66.9 |
7.9693 |
16 |
38.4 |
36.8 |
39.6 |
40.7 |
144 |
15.921 |
32 |
56.5 |
54.5 |
58.5 |
60.6 |
186 |
31.787 |
48 |
72 |
69.6 |
79 |
84.7 |
233 |
47.594 |
64 |
91 |
85 |
96 |
104.6 |
356 |
63.23 |
800
n-gram
439
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.54 |
19.75 |
22.71 |
37.22 |
39.3 |
0.998717 |
64 |
118 |
104.6 |
142.8 |
240.6 |
338 |
63.33 |
128 |
216 |
185 |
295 |
544 |
695 |
124.7 |
256 |
379.3 |
328 |
571 |
1059 |
1400 |
245.59 |
384 |
552 |
444 |
1148 |
1650 |
1879 |
362.4 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
704.2 |
160
n-gram
78
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
16.6 |
15.2 |
20.9 |
31.71 |
41.7 |
0.999427 |
8 |
33.5 |
31.1 |
42.2 |
55.3 |
84.8 |
7.9878 |
16 |
51.3 |
50.0 |
67 |
76.9 |
112.5 |
15.967 |
32 |
83.8 |
84.4 |
99.5 |
121.4 |
162 |
31.889 |
48 |
111.92 |
113.9 |
133.8 |
158.1 |
207 |
47.774 |
64 |
151 |
149.7 |
192 |
240 |
278 |
63.55 |
800
n-gram
350
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
24.2 |
20.8 |
37.4 |
38.36 |
42.4 |
0.999417 |
64 |
165 |
158 |
228 |
234.9 |
249 |
63.640 |
128 |
296 |
283 |
408 |
424 |
500 |
126.41 |
256 |
568 |
542 |
783 |
966 |
1096 |
247.8 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
526 |
160
n-gram
100
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
15.2 |
15.1 |
15.73 |
15.86 |
16.805 |
0.999 |
8 |
31.6 |
31.2 |
32.3 |
33.3 |
50.4 |
7.9875 |
16 |
57.5 |
57.8 |
59.4 |
60.4 |
78 |
15.951 |
32 |
78.4 |
78.2 |
81.0 |
82.0 |
118 |
31.861 |
48 |
103.5 |
103.5 |
106.8 |
108.3 |
135.3 |
47.713 |
64 |
133 |
132 |
139 |
142 |
221 |
63.462 |
800
n-gram
463
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
15.4 |
15.5 |
16.0 |
16.4 |
16.84 |
0.999350 |
64 |
135 |
136 |
142 |
144 |
147.1 |
63.639 |
128 |
204.2 |
209.8 |
228.1 |
232.6 |
237.8 |
126.833 |
256 |
340 |
380 |
447 |
493 |
530 |
250.95 |
384 |
555 |
613.5 |
763 |
903 |
1013 |
370.7 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
617.90 |
160
97
101
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
19.7 |
19.3 |
19.6 |
19.9 |
38.9 |
0.99876 |
n-gram |
8 |
55 |
54 |
56 |
56 |
94 |
7.9717 |
n-gram |
16 |
52.3 |
46.7 |
67.2 |
68.2 |
119 |
15.915 |
n-gram |
32 |
76.7 |
79.7 |
92.8 |
93.9 |
214 |
31.712 |
n-gram |
48 |
119.8 |
120.0 |
124.4 |
129.2 |
296.2 |
47.443 |
n-gram |
64 |
143 |
142.8 |
157 |
172 |
393 |
63.02 |
none |
1 |
19.45 |
19.19 |
19.40 |
19.48 |
32.98 |
0.998947 |
none |
8 |
52.1 |
51.9 |
55 |
55 |
85 |
7.974 |
none |
16 |
51.9 |
43.9 |
65 |
69 |
101.9 |
15.925 |
none |
32 |
74.3 |
78.4 |
91.0 |
91.6 |
194 |
31.739 |
none |
48 |
107.0 |
118.0 |
120.8 |
126 |
285 |
47.466 |
none |
64 |
132.7 |
138 |
154 |
158.8 |
374 |
63.088 |
800
400
434
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
22.3 |
21.9 |
22.8 |
30.4 |
32.2 |
0.99899 |
n-gram |
64 |
153 |
155.1 |
169 |
217.7 |
355 |
63.23 |
n-gram |
128 |
230.6 |
227.9 |
284 |
395 |
657 |
125.21 |
n-gram |
256 |
400 |
395 |
485.0 |
732 |
1790 |
245.35 |
n-gram |
384 |
716 |
644 |
762 |
1661 |
5630 |
359.0 |
none |
1 |
18.8 |
18.4 |
18.9 |
22.6 |
23.8 |
0.999267 |
none |
64 |
140 |
141 |
156 |
203 |
329 |
63.280 |
none |
128 |
212 |
166.2 |
257 |
370.3 |
625 |
125.29 |
none |
256 |
360 |
313 |
458 |
690 |
1547 |
245.81 |
none |
384 |
632 |
590 |
669 |
1543 |
4860 |
360.38 |
Language model |
# of streams |
Throughput (RTFX) |
---|---|---|
n-gram |
32 |
964 |
none |
32 |
976 |
160
n-gram
96
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.80 |
19.68 |
20.17 |
20.30 |
27.21 |
0.999 |
8 |
54.0 |
53.7 |
55.6 |
56.3 |
81.3 |
7.9842 |
16 |
51.5 |
46.1 |
68 |
70 |
105 |
15.967 |
32 |
82 |
89.8 |
93.8 |
100 |
245 |
31.8967 |
48 |
130.3 |
121.9 |
131 |
200 |
489 |
47.7950 |
64 |
259 |
147.3 |
650 |
1040 |
1640 |
63.52 |
800
n-gram
409
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
21.02 |
20.59 |
21.5 |
27.66 |
28.2 |
0.999303 |
64 |
153.3 |
148.9 |
166 |
229 |
364 |
63.661 |
128 |
225 |
216 |
264 |
426 |
697 |
126.857 |
256 |
482 |
395 |
1018 |
1751 |
2533 |
246.95 |
384 |
1112 |
641 |
3090 |
5145 |
6910 |
357.94 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
570 |
160
n-gram
87
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.96 |
20.03 |
20.32 |
20.37 |
22.49 |
0.999223 |
8 |
41.25 |
40.83 |
42.494 |
42.9 |
48.5 |
7.985 |
16 |
59.7 |
67 |
70 |
70 |
68.2 |
15.955 |
32 |
77.2 |
93 |
97.6 |
98.8 |
103.7 |
31.8790 |
48 |
117 |
129.7 |
136 |
141 |
152 |
47.713 |
64 |
157.1 |
161.0 |
176 |
182.9 |
247.9 |
63.493 |
800
n-gram
391
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.55 |
20.64 |
21.8 |
22.64 |
22.77 |
0.999217 |
64 |
164 |
162.9 |
198 |
223 |
283 |
63.496 |
128 |
272 |
270.8 |
390 |
426 |
497 |
125.95 |
256 |
467 |
455 |
820 |
970 |
1061 |
246.79 |
384 |
820 |
715 |
1620 |
1830 |
1989 |
363.9 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1100 |
160
n-gram
77
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
22.5 |
22.2 |
23.5 |
23.9 |
63.5 |
0.99794 |
8 |
47.7 |
45.7 |
51.3 |
52.4 |
106.3 |
7.9516 |
16 |
71.4 |
75.3 |
83.8 |
86.1 |
208.9 |
15.883 |
32 |
100.7 |
105.0 |
110.9 |
116 |
276 |
31.704 |
48 |
139 |
140.3 |
153 |
171 |
440 |
47.320 |
64 |
191 |
173.2 |
198.8 |
339 |
653 |
62.61 |
800
n-gram
269
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
32.1 |
31.8 |
33.9 |
57.7 |
58.7 |
0.99812 |
64 |
211 |
200 |
227.6 |
470 |
573 |
62.84 |
128 |
348 |
307 |
389 |
766 |
1004 |
123.4 |
256 |
680 |
543 |
1441 |
1910 |
2074 |
240.1 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
888 |
160
none
53
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.9 |
20.79 |
21.08 |
21.65 |
26.32 |
0.99820 |
8 |
46.9 |
46.5 |
60 |
61.0 |
69.1 |
7.952 |
16 |
67.3 |
69.7 |
78 |
80 |
107 |
15.8630 |
32 |
103 |
114 |
119.8 |
127 |
197 |
31.535 |
48 |
154 |
161 |
192 |
201.3 |
334 |
46.932 |
800
none
218
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
23.88 |
23.37 |
26.2 |
26.6 |
26.9 |
0.998203 |
64 |
204 |
207 |
254 |
342 |
414 |
62.22 |
128 |
368 |
362 |
448 |
628 |
770 |
121.54 |
none
# of streams |
Throughput (RTFX) |
---|---|
32 |
675 |
160
n-gram
102
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.1 |
19.1 |
20.07 |
20.24 |
22.65 |
0.999113 |
8 |
56.4 |
55.7 |
58.7 |
59.0 |
66 |
7.9820 |
16 |
58 |
64 |
70.7 |
71.9 |
84 |
15.946 |
32 |
84.7 |
89.7 |
92.4 |
93.5 |
144 |
31.827 |
48 |
113.7 |
117.5 |
123.1 |
128 |
185.2 |
47.632 |
64 |
135 |
139.1 |
148 |
160 |
252 |
63.423 |
800
n-gram
464
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.3 |
20.5 |
21.45 |
22.81 |
23.04 |
0.999117 |
64 |
134 |
139 |
161 |
164.9 |
171 |
63.599 |
128 |
187.9 |
181.9 |
254 |
259 |
267 |
126.68 |
256 |
324 |
318 |
448 |
485 |
528 |
251.02 |
384 |
501 |
586.0 |
659 |
848.0 |
1030 |
370.60 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1180 |
160
n-gram
94
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
20.07 |
19.53 |
20.07 |
26.54 |
32.3 |
0.998927 |
8 |
41.1 |
39.2 |
40.3 |
53.6 |
94.0 |
7.9692 |
16 |
62.2 |
62.0 |
65.2 |
71.9 |
126.3 |
15.911 |
32 |
90.2 |
89.5 |
97 |
109.2 |
279.1 |
31.706 |
48 |
128 |
118.7 |
136.9 |
223.9 |
485 |
47.391 |
64 |
198 |
144.9 |
309 |
622 |
1050 |
62.95 |
800
n-gram
321
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
21.42 |
19.95 |
27.07 |
27.18 |
27.39 |
0.999077 |
64 |
162 |
142.5 |
275.9 |
329.8 |
373.5 |
63.184 |
128 |
260 |
232 |
520.8 |
629.7 |
717.1 |
124.96 |
256 |
600 |
421 |
1429 |
1940 |
2244 |
244.18 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
686 |
160
n-gram
86
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
18.8 |
18.8 |
19.4 |
19.6 |
21.6 |
0.99858 |
8 |
60.60 |
61.15 |
63.14 |
63.85 |
65.8 |
7.96133 |
16 |
65.8 |
73.0 |
77.0 |
77.6 |
80.9 |
15.909 |
32 |
80.0 |
98 |
101 |
103 |
105 |
31.757 |
48 |
111.4 |
131 |
136.9 |
138.3 |
143 |
47.506 |
64 |
156 |
164.2 |
177 |
183 |
200 |
63.163 |
800
n-gram
395
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
19.5 |
19.7 |
20.9 |
20.31 |
20.31 |
0.99840 |
64 |
156.24 |
163.3 |
177 |
185 |
192 |
63.14 |
128 |
235 |
232 |
285 |
293 |
302 |
125.26 |
256 |
394 |
435 |
489.8 |
497.7 |
513 |
246.74 |
384 |
626.0 |
680 |
860 |
933 |
1111 |
361.3 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
952 |
100
n-gram
211
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
9.14 |
8.75 |
10.05 |
10.71 |
18.0 |
0.999320 |
8 |
12.31 |
11.40 |
13.41 |
15.20 |
40.1 |
7.9853 |
16 |
15.00 |
13.61 |
16.28 |
18.42 |
62.7 |
15.95 |
32 |
25.4 |
24.3 |
28.7 |
32.9 |
104 |
31.845 |
48 |
31.1 |
27.5 |
34.8 |
39.8 |
159.3 |
47.668 |
64 |
40.6 |
35.2 |
44.2 |
49.8 |
211 |
63.437 |
800
n-gram
806
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
14.70 |
13.97 |
20.29 |
20.48 |
21.5 |
0.999393 |
64 |
68.8 |
56.5 |
114 |
201 |
271 |
63.431 |
128 |
108.7 |
76 |
214 |
386 |
525.5 |
125.833 |
256 |
190 |
122 |
393.3 |
741 |
1085 |
247.60 |
384 |
280.0 |
169.2 |
698.8 |
1194 |
1693 |
365.46 |
512 |
388 |
213.4 |
1153 |
1674 |
2299 |
479.90 |
768 |
805 |
324 |
2624 |
3506 |
4990 |
677.3 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1600 |
100
n-gram
76
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
21.8 |
21.5 |
22.6 |
23.8 |
30.0 |
0.99904 |
8 |
35.4 |
35.0 |
38.7 |
40.2 |
58.3 |
7.9808 |
16 |
38.9 |
43.1 |
46.2 |
48.1 |
75 |
15.9453 |
32 |
62.7 |
62.1 |
69.4 |
72 |
172 |
31.819 |
48 |
94 |
83.1 |
93 |
92.2 |
464 |
47.610 |
64 |
265 |
173 |
402 |
1100 |
2050 |
63.23 |
800
n-gram
429
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
26.2 |
25.4 |
32.1 |
34.8 |
36.2 |
0.99906 |
64 |
130.0 |
116.2 |
180.3 |
267.3 |
339 |
63.297 |
128 |
223.7 |
191.5 |
339 |
514 |
654 |
125.35 |
256 |
424 |
350 |
718 |
1266 |
1660 |
245.867 |
384 |
923 |
521 |
2530 |
3553 |
4820 |
355.62 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1380 |
160
n-gram
61
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
25.90 |
25.35 |
25.68 |
25.84 |
52.59 |
0.998333 |
8 |
57 |
56.4 |
61 |
58.5 |
103.1 |
7.9651 |
16 |
60.5 |
59.2 |
65.6 |
69 |
130.0 |
15.912 |
32 |
93.1 |
89.6 |
94.4 |
97.7 |
260 |
31.697 |
48 |
139.7 |
128.9 |
137.8 |
159 |
403 |
47.342 |
800
n-gram
271
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
26.7 |
25.89 |
26.59 |
37.8 |
37.94 |
0.998783 |
64 |
170.8 |
165.3 |
182 |
257 |
403 |
63.117 |
128 |
342 |
331 |
364 |
496 |
818 |
124.56 |
256 |
736 |
630 |
713.1 |
1549 |
4761 |
241.62 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
1322 |
160
n-gram
63
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
23.22 |
23.51 |
24.12 |
24.24 |
29.11 |
0.999 |
8 |
55.5 |
56.0 |
59.9 |
61.4 |
65.1 |
7.9839 |
16 |
58.6 |
58.1 |
64.6 |
66.4 |
69.2 |
15.9643 |
32 |
86.9 |
86.8 |
91.5 |
93.3 |
105 |
31.896 |
48 |
128.9 |
128.5 |
136.2 |
138.3 |
151 |
47.7500 |
800
n-gram
298
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
26.5 |
27.2 |
27.80 |
30.84 |
31.4 |
0.999147 |
64 |
162.2 |
163 |
179.1 |
187.7 |
203 |
63.632 |
128 |
311 |
317 |
345 |
352.8 |
374 |
126.553 |
256 |
587 |
624 |
740 |
839 |
1036 |
248.83 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
942 |
160
n-gram
59
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
22.93 |
23.12 |
23.45 |
23.54 |
27.12 |
0.999117 |
8 |
57.7 |
57.7 |
62 |
63 |
68 |
7.9829 |
16 |
58.9 |
60 |
66.5 |
68.3 |
73.7 |
15.961 |
32 |
98 |
99.9 |
104 |
106 |
113 |
31.8637 |
48 |
145.8 |
147.0 |
155.2 |
161.5 |
203 |
47.631 |
800
n-gram
269
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
23.8 |
24.13 |
25.7 |
27.5 |
28.5 |
0.999127 |
64 |
188 |
185 |
273 |
293 |
358 |
63.299 |
128 |
337 |
343.0 |
570 |
609 |
660 |
125.08 |
256 |
714 |
678 |
1260 |
1319 |
1500 |
243.86 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
798 |
160
n-gram
56
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
27.96 |
27.54 |
30.14 |
31.1 |
67.8 |
0.99779 |
8 |
63.53 |
62.53 |
70.06 |
72.54 |
114.3 |
7.9561 |
16 |
67.5 |
66.1 |
76.1 |
79.5 |
198.4 |
15.891 |
32 |
101.2 |
97.6 |
107.1 |
111.8 |
262 |
31.645 |
48 |
159 |
141.5 |
156.6 |
285 |
493 |
47.18 |
800
n-gram
241
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
40.65 |
41.0 |
45.5 |
64.7 |
66.9 |
0.99786 |
64 |
234 |
209 |
342 |
466 |
600 |
62.73 |
128 |
422 |
370 |
579 |
1000 |
1250 |
123.82 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
715 |
160
n-gram
47
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
31.41 |
29.14 |
44.4 |
59.6 |
71.2 |
0.999010 |
8 |
69.3 |
63.7 |
87 |
101 |
151 |
7.9806 |
16 |
79 |
73 |
99 |
134 |
160 |
15.9523 |
32 |
134 |
130 |
159 |
184.5 |
221 |
31.832 |
800
n-gram
212
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
44.7 |
43.8 |
66.4 |
68.4 |
69 |
0.999033 |
64 |
266 |
260 |
341 |
352 |
360 |
63.39 |
128 |
505 |
484 |
690 |
743 |
930 |
124.68 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
525 |
160
n-gram
41
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
26.45 |
26.64 |
27.1 |
27.3 |
30.63 |
0.998817 |
8 |
60.0 |
59.7 |
66.3 |
67.4 |
72.6 |
7.9765 |
16 |
73 |
77.9 |
82 |
84 |
118 |
15.925 |
32 |
141 |
143.30 |
148.54 |
153.6 |
250 |
31.6993 |
800
n-gram
196
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
30.41 |
30.72 |
32.2 |
32.84 |
32.88 |
0.99874999999999992 |
64 |
263 |
270.2 |
278.9 |
287 |
292 |
63.303 |
128 |
454 |
431 |
537 |
639 |
692 |
124.97 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
648.6 |
160
39
40
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
53.4 |
53.10 |
54.05 |
54.31 |
111 |
0.99649 |
n-gram |
8 |
82 |
72 |
106 |
112 |
156.0 |
7.955 |
n-gram |
16 |
104.1 |
108.7 |
114.5 |
116.0 |
226.6 |
15.8670 |
n-gram |
32 |
239 |
217 |
310 |
421 |
610 |
31.505 |
none |
1 |
49.40 |
49.755 |
49.94 |
49.97 |
95.3 |
0.99702 |
none |
8 |
64.5 |
65.3 |
65.7 |
66.5 |
123 |
7.9634 |
none |
16 |
102.3 |
110 |
113 |
114 |
207.6 |
15.8743 |
none |
32 |
191.5 |
190.8 |
235.9 |
340 |
588 |
31.525 |
800
177
183
Language model |
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
|||
n-gram |
1 |
59.1 |
58.71 |
60.24 |
77.1 |
78.1 |
0.997583 |
n-gram |
64 |
310 |
315 |
332 |
431 |
586 |
62.75 |
n-gram |
128 |
505 |
463 |
583 |
760 |
1513 |
123.633 |
none |
1 |
50.7 |
49.90 |
50.6 |
58.26 |
58.53 |
0.998160 |
none |
64 |
285 |
293.0 |
302 |
375 |
518 |
62.911 |
none |
128 |
464 |
382 |
545 |
729 |
1470 |
123.710 |
Language model |
# of streams |
Throughput (RTFX) |
---|---|---|
n-gram |
32 |
488 |
none |
32 |
484 |
160
n-gram
38
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
51.3 |
50.8 |
53.01 |
53.38 |
74.9 |
0.99838 |
8 |
64.9 |
65.4 |
71.3 |
73.6 |
120.2 |
7.976 |
16 |
108.4 |
116 |
123 |
133 |
231 |
15.933 |
32 |
408 |
218.2 |
950 |
1370 |
2010 |
31.535 |
800
n-gram
171
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
55.1 |
53.5 |
55.93 |
76.5 |
76.10 |
0.998263 |
64 |
294.0 |
299.1 |
318 |
445 |
609.6 |
63.339 |
128 |
600 |
538.6 |
1067 |
1290 |
1746 |
123.49 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
281 |
160
n-gram
37
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
52.15 |
52.52 |
53.64 |
53.74 |
60.42 |
0.998037 |
8 |
77.9 |
78.1 |
97.55 |
98.7 |
101.3 |
7.9739 |
16 |
106 |
115 |
121 |
122 |
135 |
15.909 |
32 |
246.7 |
234.9 |
381 |
397 |
424 |
31.657 |
800
n-gram
169
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
54.2 |
54.87 |
57.7 |
60.94 |
61.03 |
0.998027 |
64 |
338 |
333 |
418 |
514 |
540 |
62.77 |
128 |
574.9 |
562 |
1050 |
1100 |
1149 |
122.98 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
481 |
160
n-gram
36
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
58.98 |
59.07 |
62.89 |
63.39 |
111.9 |
0.99638 |
8 |
85.8 |
77.23 |
108.8 |
111.3 |
143 |
7.9399 |
16 |
115.2 |
121.2 |
137 |
141 |
262 |
15.864 |
32 |
274 |
236.1 |
402 |
444 |
659 |
31.39 |
800
n-gram
139
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
76.2 |
77.8 |
81.3 |
105.0 |
109.2 |
0.99654 |
64 |
391.9 |
375 |
480 |
640 |
807 |
62.31 |
128 |
741 |
618 |
1450 |
1736 |
2220 |
120.9 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
495 |
160
none
32
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
42.5 |
42.2 |
42.6 |
43.9 |
59.9 |
0.996083 |
8 |
64.4 |
65.1 |
78.3 |
83 |
117 |
7.929 |
16 |
90.1 |
95.55 |
103.7 |
110 |
158.9 |
15.7900 |
none
# of streams |
Throughput (RTFX) |
---|---|
32 |
348 |
160
n-gram
40
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
51.90 |
51.4 |
53.14 |
54.2 |
93.1 |
0.99761 |
8 |
86 |
70.7 |
106.4 |
111.4 |
120 |
7.9699 |
16 |
95.9 |
114 |
122 |
121.6 |
169 |
15.898 |
32 |
242 |
237 |
302 |
366 |
441 |
31.583 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
523 |
160
n-gram
37
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
51.6 |
50.7 |
52.64 |
73.58 |
109.4 |
0.99643 |
8 |
87.9 |
94 |
107 |
107.4 |
153.4 |
7.9542 |
16 |
106 |
85.6 |
116.6 |
148 |
293 |
15.845 |
32 |
359 |
282 |
575 |
790 |
1330 |
31.3700 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
322 |
160
n-gram
38
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
52.2 |
52.1 |
54.12 |
54.28 |
60.2 |
0.99625 |
8 |
84.0 |
70.6 |
106.78 |
114.2 |
121 |
7.9374 |
16 |
103.9 |
88.9 |
119.9 |
121.18 |
130 |
15.8593 |
32 |
267 |
259 |
322 |
389 |
397.3 |
31.37 |
800
n-gram
173
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
55.1 |
55.79 |
58.75 |
59.3 |
59.3 |
0.995667 |
64 |
315 |
326 |
348 |
352 |
359 |
62.371 |
128 |
488.5 |
484 |
581 |
589 |
606.7 |
122.52 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
468 |
100
n-gram
121
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
16.67 |
15.80 |
18.98 |
20.8 |
35.3 |
0.998667 |
8 |
26.989 |
25.14 |
31.0 |
35.6 |
79.2 |
7.9721 |
16 |
37.7 |
35.8 |
47.3 |
54 |
122.2 |
15.913 |
32 |
47.7 |
44.6 |
58.9 |
68.1 |
187.6 |
31.763 |
48 |
58.4 |
53.8 |
70 |
82 |
262 |
47.570 |
64 |
73 |
66 |
80 |
96 |
367 |
63.295 |
800
n-gram
469
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
38.7 |
39.19 |
47.91 |
48.8 |
52 |
0.99847 |
64 |
158 |
144 |
222 |
328 |
413 |
63.17 |
128 |
216 |
187 |
332 |
521 |
675 |
125.29 |
256 |
369 |
308 |
654 |
1150 |
1479 |
245.8 |
384 |
621 |
446 |
1452 |
2110 |
2950 |
360.27 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
957 |
100
n-gram
38
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
47.33 |
47.17 |
49.9 |
53.4 |
65.31 |
0.998 |
8 |
46.1 |
46.3 |
50.9 |
55.10 |
82.5 |
7.9744 |
16 |
73 |
71 |
71.9 |
81 |
134.2 |
15.9247 |
32 |
321 |
264 |
387 |
792 |
1600 |
31.676 |
800
n-gram
246
# of streams |
Latency (ms) |
Throughput (RTFX) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
1 |
69.74 |
69.0 |
77.7 |
84.9 |
85.7 |
0.997620 |
64 |
217.7 |
208.3 |
281 |
367 |
446 |
63.088 |
128 |
398.7 |
369.0 |
512 |
698 |
1179 |
124.54 |
n-gram
# of streams |
Throughput (RTFX) |
---|---|
32 |
807 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
17.26 |
17.31 |
17.83 |
18.10 |
20.44 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
17.04 |
16.98 |
17.80 |
18.31 |
20.46 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
17.70 |
17.72 |
18.40 |
18.78 |
20.12 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
19.26 |
18.15 |
21.05 |
23.06 |
48.49 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
18.09 |
17.95 |
18.28 |
18.52 |
21.19 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
21.71 |
21.64 |
22.76 |
23.19 |
25.20 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
24.73 |
24.42 |
25.69 |
26.06 |
31.10 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
24.48 |
24.29 |
25.53 |
26.29 |
29.02 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
25.10 |
25.03 |
26.16 |
26.60 |
29.13 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
24.48 |
24.38 |
25.25 |
25.76 |
28.82 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
24.24 |
24.06 |
24.38 |
24.56 |
29.70 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
24.66 |
24.70 |
25.70 |
26.10 |
29.82 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
24.54 |
24.34 |
25.57 |
26.05 |
29.21 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
25.04 |
24.89 |
26.14 |
27.01 |
29.90 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
28.76 |
28.75 |
29.78 |
30.36 |
34.49 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
28.55 |
28.42 |
29.82 |
30.25 |
33.60 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
#DIV/0! |
#DIV/0! |
#DIV/0! |
#DIV/0! |
#DIV/0! |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
31.62 |
29.94 |
34.26 |
36.77 |
74.98 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
29.85 |
29.57 |
30.08 |
30.38 |
40.87 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
34.94 |
34.82 |
36.50 |
37.56 |
40.45 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
37.79 |
37.32 |
39.27 |
40.14 |
48.10 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
37.00 |
36.72 |
38.67 |
39.82 |
43.10 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
#DIV/0! |
#DIV/0! |
#DIV/0! |
#DIV/0! |
#DIV/0! |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
37.10 |
36.88 |
38.52 |
39.20 |
42.59 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
37.25 |
26.95 |
37.36 |
37.66 |
47.64 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
40.35 |
39.15 |
45.41 |
49.94 |
69.77 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
36.72 |
36.26 |
38.43 |
39.69 |
45.42 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
37.99 |
37.71 |
39.60 |
41.36 |
46.14 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
9.25 |
9.29 |
9.59 |
9.71 |
10.96 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
9.75 |
9.27 |
9.60 |
9.92 |
10.43 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
9.83 |
9.43 |
9.65 |
9.85 |
10.44 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
10.73 |
9.79 |
12.18 |
13.33 |
30.67 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
10.30 |
10.27 |
10.39 |
10.48 |
11.15 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
11.25 |
11.20 |
11.77 |
12.07 |
12.74 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
12.59 |
12.41 |
13.32 |
13.48 |
17.16 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
11.68 |
11.61 |
12.28 |
12.82 |
14.09 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
12.21 |
12.19 |
12.85 |
13.14 |
14.48 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
11.96 |
11.90 |
12.48 |
12.73 |
13.88 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
13.58 |
13.52 |
13.73 |
13.87 |
14.89 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
12.14 |
12.18 |
12.83 |
13.04 |
14.29 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
11.52 |
11.41 |
12.19 |
12.58 |
13.90 |
160
Language model |
# of streams |
Latency (ms) |
||||
---|---|---|---|---|---|---|
avg |
p50 |
p90 |
p95 |
p99 |
||
n-gram |
1 |
14.07 |
13.98 |
14.72 |
15.19 |
17.02 |
Hardware Specifications#
GPU |
|
---|---|
NVIDIA DGX A100 40 GB |
|
CPU |
|
Model |
AMD EPYC 7742 64-Core Processor |
Thread(s) per core |
2 |
Socket(s) |
2 |
Core(s) per socket |
64 |
NUMA node(s) |
8 |
Frequency boost |
enabled |
CPU max MHz |
2250 |
CPU min MHz |
1500 |
RAM |
|
Model |
Micron DDR4 36ASF8G72PZ-3G2B2 3200MHz |
Configured Memory Speed |
2933 MT/s |
RAM Size |
32x64GB (2048GB Total) |
GPU |
|
---|---|
NVIDIA A30 |
|
CPU |
|
Model |
AMD EPYC 7742 64-Core Processor |
Thread(s) per core |
1 |
Socket(s) |
2 |
Core(s) per socket |
64 |
NUMA node(s) |
2 |
Frequency boost |
disabled |
CPU max MHz |
2250.0000 |
CPU min MHz |
1500.0000 |
RAM |
|
Model |
Samsung DDR4 M393A4K40DB3-CWE 3200MHz |
Configured Memory Speed |
3200 MT/s |
RAM Size |
32x64GB (2048GB Total) |
GPU |
|
---|---|
NVIDIA A10 |
|
CPU |
|
Model |
AMD EPYC 7763 64-Core Processor |
Thread(s) per core |
1 |
Socket(s) |
2 |
Core(s) per socket |
64 |
NUMA node(s) |
8 |
Frequency boost |
enabled |
CPU max MHz |
2450 |
CPU min MHz |
1500 |
RAM |
|
Model |
Samsung DDR4 M393A4K40DB3-CWE 3200 MHz |
Configured Memory Speed |
3200 MT/s |
RAM Size |
16x32GB (512GB Total) |
GPU |
|
---|---|
NVIDIA V100 SXM2 16 GB |
|
CPU |
|
Model |
Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz |
Thread(s) per core |
2 |
Socket(s) |
2 |
Core(s) per socket |
20 |
NUMA node(s) |
2 |
CPU max MHz |
3600 |
CPU min MHz |
1200 |
RAM |
|
Model |
Micron DDR4 36ASF4G72PZ-2G6D1 2667MHz |
Configured Memory Speed |
2133 MT/s |
RAM Size |
16x32GB (512GB Total) |
GPU |
|
---|---|
NVIDIA T4 |
|
CPU |
|
Model |
Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz |
Thread(s) per core |
2 |
Socket(s) |
2 |
Core(s) per socket |
18 |
NUMA node(s) |
2 |
CPU max MHz |
3900 |
CPU min MHz |
1000 |
RAM |
|
Model |
Samsung DDR4 M393A2K43BB1-CTD 2666MHz |
Configured Memory Speed |
2666 MT/s |
RAM Size |
24x16GB (384GB Total) |