Is this page helpful?

ASR NIM Performance#

This page provides latency and throughput benchmarks for the NVIDIA ASR NIM microservice in streaming and offline configurations across supported GPUs.

Evaluation Process#

The performance evaluation is done for the two different inference modes: streaming and offline.

Streaming Mode#

Streaming benchmarks use the riva_streaming_asr_client with the --simulate_realtime flag to simulate real-time microphone transcription. The client and server process audio chunks of the same duration. Each stream performs three iterations over a sample audio file (1272-135031-0000.wav) from the LibriSpeech dev-clean dataset. Refer to the Results section for chunk size values.

The source code for riva_streaming_asr_client is available at Riva C++ Clients.

The following command measures streaming performance:

riva_streaming_asr_client \
   --chunk_duration_ms=<chunk_duration> \
   --simulate_realtime=true \
   --automatic_punctuation=true \
   --num_parallel_requests=<num_streams> \
   --word_time_offsets=false \
   --print_transcripts=false \
   --interim_results=false \
   --num_iterations=<3*num_streams> \
   --audio_file=1272-135031-0000.wav \
   --output_filename=/tmp/output.json

The riva_streaming_asr_client returns three latency measurements:

  • intermediate latency: Latency of responses with is_final == false.

  • final latency: Latency of responses with is_final == true.

  • latency: Overall latency of all responses. This value is reported in the results tables.

The following diagram shows how these latencies are measured:

Schematic Diagram of Latencies Measured by Riva Streaming ASR Client

Offline Mode#

The following command measures maximum throughput in offline mode:

riva_asr_client \
   --automatic_punctuation=true \
   --num_parallel_requests=32 \
   --word_time_offsets=false \
   --print_transcripts=false \
   --num_iterations=96 \
   --audio_file=1272-135031-0000x5.wav \
   --output_filename=/tmp/output.json

The file 1272-135031-0000x5.wav is the 1272-135031-0000.wav audio file concatenated five times. The source code for riva_asr_client is available at Riva C++ Clients.

Results#

The following tables report latency and throughput for streaming and offline configurations. Throughput is measured in RTFX (duration of audio transcribed divided by computation time).

Note

All values are averages over three trials, rounded to the last significant digit based on standard deviation. If a standard deviation is less than 0.001 of the average, the value is rounded as if the standard deviation equals 0.001 of the average.

For the hardware used in these measurements, refer to the Hardware Specifications section.

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 270

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

12.439

10.388

11.524

12.54

30.242

0.99949

8

13.006

12.508

14.673

17.203

29.539

7.9929

16

18.138

17.06

24.885

27.922

49.825

15.975

32

23.093

20.141

29.905

30.991

76.264

31.915

48

28.666

29.63

33.027

34.111

101.76

47.834

64

32.012

32.449

35.855

37.779

136.06

63.719
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1240

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

14.088

12.616

17.525

20.743

51.894

0.9994

64

42.41

37.123

43.153

157.71

163.47

63.68

128

61.41

49.455

61.86

197.73

307.18

126.82

256

93.439

67.938

98.617

315.2

558.71

251.39

384

123.79

93.576

124.35

472.36

848.59

373.93

512

166.85

117.95

318.19

615.9

1141.7

494.12
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

353.62

False

32

3707.4

True

32

170
Chunk size (ms): 320
Language model: n-gram
Maximum effective # of streams with n-gram language model: 270

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

16.946

14.233

17.459

18.954

21.387

0.99936

8

19.451

18.632

22.736

26.336

32.004

7.9924

16

24.811

23.88

28.443

32.162

42.753

15.978

32

33.166

30.537

43.692

47.19

68.485

31.929

48

44.522

49.317

57.667

60.352

93.513

47.855

64

55.794

60.906

71.632

74.103

117.78

63.755
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1240

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

19.719

19.153

22.404

25.33

53.192

0.99929

64

67.47

73.823

84.609

89.695

91.372

63.803

128

116.36

122.78

146.72

152.17

173.31

127.31

256

174.21

179.51

223.23

242.4

270.84

253.54

384

225.73

208.54

317.68

323.05

345.72

379.42

512

281.13

299.92

406.08

416.05

517.49

503.1
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

335.83

False

32

3890.1
Chunk size (ms): 160
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

27.7

26.0

27.7

28.5

42.8

1.0

8

33.1

32.9

35.1

35.9

54.1

8.0

16

43.4

42.7

45.7

57.2

73.9

16.0

32

59.7

48.4

78.1

80.1

105.4

31.9

48

99.1

106.5

110.5

112.2

187.5

47.7
Chunk size (ms): 960
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

23.7

22.9

26.3

27.6

50.4

1.0

64

135.7

160.3

167.9

171.1

174.8

63.7

128

273.6

300.2

314.6

319.5

344.2

126.8
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

224.8

False

32

1223.7
Chunk size (ms): 160
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

34.3

33.8

35.5

36.1

61.2

1.0

True

1

38.1

34.8

48.5

50.0

93.0

1.0

False

8

41.2

40.7

43.1

43.9

76.4

8.0

True

8

53.6

41.4

89.8

96.9

165.4

8.0

False

16

52.4

51.1

53.9

70.7

104.1

15.9

True

16

70.9

51.6

114.8

129.8

257.8

15.9

False

32

78.4

64.4

102.5

105.8

145.8

31.8

True

32

115.5

102.1

201.6

217.2

394.5

31.6

False

48

105.5

124.7

132.7

136.8

174.0

47.6

True

48

169.7

141.4

258.0

287.1

518.0

47.3
Chunk size (ms): 960
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

62.0

61.3

64.7

65.6

92.2

1.0

True

1

93.9

76.6

100.4

101.0

538.6

1.0

False

64

230.0

269.0

275.3

278.4

280.1

63.4

True

64

388.3

425.9

495.3

510.8

525.9

63.2

False

128

366.7

398.8

416.5

429.6

446.6

126.2

True

128

600.0

621.0

644.7

900.0

957.3

124.4
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

21.0

False

32

357.2

True

32

270.1
Chunk size (ms): 160
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

23.02

23.176

28.694

29.829

44.012

0.99863

8

34.593

33.966

40.212

45.607

94.882

7.9646

16

42.333

41.683

50.64

58.927

93.028

15.953

32

55.452

51.111

76.248

82.525

129.15

31.828

48

72.80

175.236

94.222

106.1

223.45

47.592

64

97.943

100.13

116.06

126.09

240.68

63.512
Chunk size (ms): 960
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

29.309

29.149

36.643

41.075

47.067

0.99879

64

114.55

116.94

159.24

177.71

189.42

63.655

128

170.28

173.34

217

220.87

306.83

126.76

256

265.46

262.12

374.34

445.03

610.07

251.12

384

322.1

300.5

478.52

627.84

962.12

374.99

512

437.49

385.16

733.84

1084.8

1529.9

493.42
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

166.71

False

32

505.29
Chunk size (ms): 160
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

17.854

15.285

20.322

21.213

28.771

0.99935

8

24.344

22.916

30.133

38.193

53.096

7.9913

16

33.628

31.514

39.454

61.332

77.594

15.975

32

51.488

51.29

60.141

99.328

125.21

31.915

48

66.051

66.906

78.255

106.77

150.16

47.831

64

70.315

75.312

85.973

123.24

183.86

63.743
Chunk size (ms): 960
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

21.612

20.434

25.3

26.234

62.83

26.234

64

95.964

87.699

181.02

183.42

188.01

189.01

128

174.07

152.57

280.41

346.19

356.8

189.01

256

281.56

249.97

523.14

594.08

682.59

700.86

384

392.68

336.15

758.24

870.87

1002.9

1033.1

512

540.34

437.2

1118.3

1210.2

1351.9

1424.3
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

337.94

False

32

3229.5
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

14.5

13.6

14.8

15.3

22.0

1.0

8

16.7

16.4

17.7

18.6

33.8

8.0

16

19.4

18.6

20.3

26.8

49.1

16.0

32

25.1

26.8

31.0

31.8

75.8

31.9

48

30.0

31.7

35.7

39.1

99.0

47.8

64

36.1

38.2

41.8

43.3

138.0

63.7
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

19.5

17.4

23.6

26.4

80.3

1.0

64

48.7

45.3

52.4

154.2

164.1

63.7

128

65.8

54.4

74.6

198.1

285.4

126.8

256

99.9

78.1

113.7

356.7

554.2

251.6

384

130.2

101.0

150.3

488.8

788.8

374.2

512

178.8

130.8

316.2

638.8

1106.0

494.6
Language model: n-gram

# of streams

Throughput (RTFX)

1

303.6

32

3331.5
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

38.3

18.6

29.5

87.6

536.4

1.0

8

25.4

22.3

33.9

34.4

81.2

7.9

16

37.7

35.7

56.4

58.3

123.2

15.8

32

38.1

37.7

62.2

65.1

136.0

31.5

48

46.1

46.1

70.2

73.0

142.9

47.1

64

58.5

57.0

84.7

87.0

275.5

62.4

128

75.2

64.1

114.9

120.0

415.4

122.1
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

66.0

40.9

49.0

338.8

338.8

1.0

64

111.4

117.9

124.8

273.5

275.2

62.4

128

152.1

144.6

166.0

363.3

479.8

122.5
Language model: n-gram

# of streams

Throughput (RTFX)

1

50.4

32

605.5

64

782.7

128

1004.9

# of streams

Throughput (RTFX)

Average Latency (ms)

1

208.9

272.85

32

2210.3

745.51

64

2601

810.1
Chunk size (ms): 320
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

1.0

16

15.7

32

31.0

64

60.9

128

117.3
Chunk size (ms): 1600
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

1.0

64

61.8

128

121.3
Language model: n-gram

# of streams

Throughput (RTFX)

1

28.9

32

615.3

32

782.2

64

803.9

128

1170.7
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 355

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

10

9.9

11.3

12

40

1

8

12.6

12

13.4

17

31

8

16

17

15

22

25

40

15.98

32

23

23

31

33

50

31.94

48

29

28

40

41

70

47.9

64

33.6

38

45

47

70

63.9

128

49

47

64

67

150

127.6

256

84

75

107

126

391

255
Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1400

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

14

11

20

40

80

1

64

39

40

55

80

110

63.9

128

58

50

75

150

202

127.6

256

90

80

115

240

380

255

384

120

107

155

316

530

381.4

512

149

130

196

400

700

508

768

258

200

630

680

1280

756

1024

420

263

1280

1350

1900

992
Language model: n-gram

# of streams

Throughput (RTFX)

32

467
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

90

False

32

370
Language model: none

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

110

False

32

1255
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

6.6031

6.6753

7.1572

7.2425

8.7497

0.99967

False

8

6.8314

6.8296

7.4957

7.6497

9.5597

7.9973

False

16

6.8778

6.8022

7.5539

7.8327

11.321

15.995

False

32

8.6936

8.3926

11.355

11.705

13.094

31.983

False

64

10.518

10.724

14

14.547

19.139

63.945
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

7.1939

7.2401

7.8403

7.9414

9.1036

0.99966

False

64

10.842

11.489

13.368

13.804

15.752

63.953

False

128

13.14

13.316

16.333

16.98

22.896

127.86

False

512

19.075

17.745

26.808

28.797

56.3

510.24

False

768

24.194

22.802

35.175

38.299

86.369

764.52

False

1024

28.018

26.633

42.099

45.898

90.656

1018

False

1200

29.466

27.126

43.277

50.569

97.406

1192

False

1400

31.977

27.85

48.475

54.571

143.6

1389.3

False

1536

34.218

28.822

50.877

59.797

166.23

1522.8

False

1600

33.047

27.295

48.753

63.013

194.59

1585.7

False

1800

37.174

29.501

53.728

83.721

214.75

1782.4

False

1900

42.398

37.688

60.881

78.298

202

1880.5

False

2048

63.832

54.595

99.731

149.55

296.13

2024
Chunk size (ms): 160
Batch size: 512
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

10.062

10.061

10.67

10.772

13.491

0.99956

False

128

17.586

18.082

21.241

21.862

27.421

127.83

False

512

22.845

22.453

28.862

30.552

49.08

510.44

False

1024

27.982

26.1

35.974

39.433

87.307

1018.5

False

1536

31.533

25.762

40.77

52.357

180.55

1525

False

2048

41.581

30.414

50.889

98.42

323.09

2026.4

False

2304

49.183

36.175

58.137

128.45

385.72

2277.7

False

2560

60.855

42.53

67.996

194.11

480.4

2524

False

2660

74.877

45.396

136.52

319.67

556.82

2621.8

False

2816

161.83

63.055

391.83

467.75

580.02

2761.7

False

2900

93.014

54.012

197.19

400.28

648.21

2853.9
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

6.6522

6.6362

7.2231

7.3482

9.0424

0.99967

False

8

6.7588

6.744

7.2969

7.4898

9.7917

7.9973

False

16

6.9039

6.8165

7.4692

7.7616

11.382

15.995

False

32

8.7274

8.2067

11.294

11.613

12.949

31.984

False

64

10.303

10.607

13.751

14.315

18.406

63.948
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

7.3295

7.3418

7.9362

8.0996

9.8249

0.99968

False

64

11.111

11.83

13.6

14.014

17.208

63.951

False

128

12.429

12.464

15.598

16.742

20.562

127.85

False

512

18.012

17.071

24.307

27.12

48.71

510.27

False

768

23.926

24.118

32.748

36.113

70.207

764.6

False

1024

28.547

27.179

40.907

43.547

90.518

1017.8

False

1200

27.017

25.434

38.775

42.313

101

1192.3

False

1400

29.96

27.38

41.764

47.752

117.03

1389.5

False

1536

35.57

31.415

50.083

58.727

161.09

1523.3

False

1600

35.157

30.459

49.168

56.036

174.68

1585.8

False

1800

34.807

28.049

51.246

67.028

184.58

1782.8

False

1900

39.375

30.168

52.243

108.06

275.61

1880.6

False

2048

50.967

45.068

70.391

114.85

242.16

2024.4
Chunk size (ms): 160
Batch size: 512
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

10.058

10.094

10.613

10.71

13.559

0.99956

False

128

17.415

17.636

20.988

21.702

27.874

127.83

False

512

21.665

21.12

26.474

27.92

48.717

510.49

False

1024

25.981

24.617

31.28

33.962

82.857

1019.1

False

1536

29.994

24.878

37.096

47.647

168.07

1525.4

False

2048

41.861

32.613

47.422

94.576

317.09

2029.3

False

2304

46.423

31.994

51.149

140.62

390.84

2277.5

False

2560

54.391

38.062

60.824

162.55

438.38

2526.6

False

2660

70.172

44.921

123.04

273.59

502.72

2621.1

False

2816

71.851

45.251

112.57

274.47

537.44

2772.8

False

2900

83.945

50.335

126.97

301.26

596.5

2855.9
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 179

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

17.79

15.812

22.171

22.66

24.527

0.99925

8

19.619

18.702

20.283

21.283

49.858

7.9866

16

24.347

22.816

24.601

30.805

83.174

15.958

32

32.883

30.65

40.314

40.992

129.39

31.856

48

43.084

44.219

50.994

56.952

210.66

47.689

64

53.643

53.416

61.031

97.948

264.43

63.476
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 810

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

20.624

19.301

26.012

28.626

50.408

0.99918

64

73.596

71.06

84.034

234.8

251.47

63.497

128

123.56

110.62

139.56

300.04

449.06

126.28

256

188.12

162.21

200.33

538

814.35

249.61

384

268.43

198.76

527.84

786.32

1372.3

369.42

512

405.24

287.28

1347.4

1439.1

2252.5

486.61
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

318.26

False

32

2085

True

32

125
Chunk size (ms): 320
Language model: n-gram
Maximum effective # of streams with n-gram language model: 179

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

25.742

25.178

29.273

30.679

40.96

0.99891

8

37.458

36.717

43.875

45.592

57.38

7.9865

16

46.788

45.738

51.555

60.965

75.74

15.963

32

64.08

57.471

84.993

89.653

128.29

31.873

48

85.545

96.194

111.54

117.86

176.5

47.714

64

93.02

104.95

116

124.89

195.03

63.61
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 810

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

21.451

20.791

23.836

24.61

53.358

0.99922

64

91.55

103.46

124.84

126.52

134.28

63.575

128

177.23

190.8

213.12

218.6

244.28

127.01

256

279.71

279.51

358.52

371.5

449.47

252.36

384

386.16

389.76

521.21

556.27

722.57

375.28

512

492.63

496.77

691.83

793.61

1101.9

494.73
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

338.78

False

32

3041.9
Chunk size (ms): 160
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

41.4

41.0

43.1

43.9

65.2

1.0

8

69.5

68.9

73.8

76.8

116.3

8.0

16

84.7

80.4

108.1

113.5

149.9

15.9

32

138.2

147.3

172.9

180.1

232.4

31.7

48

2610.6

2456.5

4743.8

4941.9

6120.2

41.5
Chunk size (ms): 960
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

32.0

32.0

33.2

34.9

35.6

1.0

64

263.5

310.4

324.9

326.4

342.2

63.4

128

562.0

591.7

646.0

829.8

835.5

124.8
Language model: n-gram

# of streams

Throughput (RTFX)

1

151.5

32

599.4
Chunk size (ms): 160
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

51.9

51.2

53.3

54.1

91.2

1.0

True

1

57.0

51.5

74.3

75.6

134.2

1.0

False

8

78.6

77.6

82.8

84.0

144.3

8.0

True

8

92.7

80.3

127.8

131.8

246.6

7.9

False

16

85.0

83.8

86.4

87.3

165.2

15.9

True

16

107.9

85.5

161.3

164.5

350.9

15.8

False

32

147.0

149.3

176.1

184.5

295.1

31.7

True

32

273.1

241.1

415.1

505.6

817.4

31.2
Chunk size (ms): 960
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

74.0

73.7

75.3

77.5

94.4

1.0

True

1

108.0

96.6

100.2

111.7

473.1

1.0

False

64

366.4

427.7

438.1

447.2

456.1

63.1

True

64

541.1

604.0

658.0

803.4

833.1

62.4
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

18.7

False

32

252.4

True

32

148.7
Chunk size (ms): 160
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

30.948

31.181

34.543

36.056

47.975

0.99827

8

47.991

48.392

53.894

55.871

87.543

7.978

16

61.284

61.356

68.72

76.258

118.18

15.923

32

75.633

74.065

95.78

102.73

155.86

31.809

48

91.854

99.673

111.41

113.95

255.06

47.621

64

114.38

126.17

135.55

139.42

321.86

63.361
Chunk size (ms): 960
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

26.371

26.905

29.871

34.808

35.852

0.9987

64

116.95

132.38

156.75

165.34

170.12

63.681

128

227.25

232.36

279.57

295.28

372.04

126.56

256

351

363.49

448.89

506.45

769.04

249.55

384

451.4

451.33

622.64

676.79

935.07

372.85

512

579.83

578.14

838.03

1041.4

1447.4

489.7
Language model: n-gram

# of streams

Throughput (RTFX)

1

213.96

32

1021
Chunk size (ms): 160
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

25.898

25.159

29.92

31.529

38.217

0.99898

8

39.652

38.118

48.426

56.406

75.964

7.986

16

51.214

48.284

58.617

78.85

103.47

15.963

32

62.971

65.896

79.542

102.56

123.49

31.882

48

82.938

89.665

102.51

151.8

176.93

47.767

64

107.35

113.37

128.69

197.71

246.97

63.598
Chunk size (ms): 960
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

23.143

22.251

26.383

26.517

62.112

0.9993

64

123.91

120.69

209.79

221.48

222.74

63.746

128

250.18

224.35

404.34

470.45

495.53

127

256

424.4

391.15

726.09

848.57

940.01

250.66

384

644.41

550.55

1274

1344.7

1493.9

369.39

512

882.82

803.01

1746.6

1874.4

1955.3

486.89
Language model: n-gram

# of streams

Throughput (RTFX)

1

299.83

32

2790.2
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

26.7

26.5

27.8

28.3

46.2

1.0

8

32.1

31.1

32.5

33.3

75.2

8.0

16

39.2

35.2

53.3

54.4

102.4

15.9

32

47.4

38.6

61.9

64.3

140.2

31.8

48

58.2

64.3

68.8

74.1

188.8

47.7

64

73.2

77.7

81.6

119.6

282.5

63.5
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

30.4

30.0

31.3

37.2

37.8

1.0

64

100.0

99.9

117.6

256.4

276.0

63.5

128

137.8

109.5

164.0

355.9

468.1

126.2

256

218.3

175.4

281.1

611.2

892.1

249.0

384

350.6

280.9

838.8

937.2

1633.7

367.5

512

624.9

365.5

1666.0

2471.5

2800.5

481.6
Language model: n-gram

# of streams

Throughput (RTFX)

1

166.2

32

1555.0
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

70.0

38.8

51.9

285.2

693.8

1.0

8

42.7

37.7

53.4

54.2

134.4

7.9

16

50.4

44.2

64.2

67.1

193.3

15.7

32

61.7

56.4

95.7

98.9

207.7

31.2

48

71.0

69.2

109.0

115.3

278.3

46.5

64

88.1

93.1

129.3

136.4

341.7

61.3

128

162.9

149.1

243.1

252.6

769.8

117.8
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

83.9

57.0

67.5

382.3

382.3

1.0

64

165.8

180.7

206.0

386.2

387.1

61.8

128

240.4

229.1

293.1

566.6

722.3

120.0
Language model: n-gram

# of streams

Throughput (RTFX)

1

31.2

32

431.1

64

536.7

128

633.3

# of streams

Throughput (RTFX)

Average Latency (ms)

1

128.5

433.0

32

1326.0

1268.77
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 233

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

13

11.8

12.8

14

40

1

8

17.6

16.8

18.5

22

39

8

16

22.5

21.3

25

31

60.3

15.98

32

32.4

35

42

46

70

31.93

48

41

40

58

59

100

47.9

64

46

50

64

66

100

63.8

128

73

66

94

97

220

127.5
Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 980

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

16

13

20

40

80

1

64

60

60

80

110

180

63.8

128

90

80

110

230

300

127.5

256

133.3

120

174

340

530

254

384

183

166

245

430

800

380

512

260

223

510

600

1200

505

768

535

354

1500

1640

2150

739

1024

940

600

2300

2570

2930

960
Language model: n-gram

# of streams

Throughput (RTFX)

32

460
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

60

False

32

234
Language model: none

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

108.04

False

32

827.71
Chunk size (ms): 320
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

1.0

16

15.4

32

30.3

64

58.8

128

107.7
Chunk size (ms): 1600
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

1.0

64

60.7

128

117.3
Language model: n-gram

# of streams

Throughput (RTFX)

1

23.0

32

445.6

32

545.9

64

655.7

128

904.6
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

9.4185

9.4797

10.227

10.361

12.929

0.99957

False

8

10.253

9.649

10.39

10.711

13.52

7.9963

False

16

10.156

9.8396

10.933

15.996

17.537

15.989

False

32

13.388

13.85

17.733

18.169

22.81

31.973

False

64

16.517

17.181

23.175

23.99

31.369

63.912
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

10.48

10.536

11.305

11.394

14.596

0.9995

False

64

19.656

19.818

22.768

23.624

39.067

63.924

False

128

22.949

23.713

27.031

27.79

37.807

127.78

False

512

36.519

28.835

46.102

59.58

225.28

509.51

False

768

38.891

37.225

57.876

61.546

107.2

762.88

False

1024

56.52

41.354

107.75

173.77

283.01

1015.3

False

1200

66.418

50.971

133.97

204.05

271.2

1187.1

False

1400

141.31

118.77

287.49

330.25

467.44

1371.6

False

1536

1288.4

1199.9

2414.4

2625.2

2889

1401.3
Chunk size (ms): 160
Batch size: 512
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

18.684

18.703

19.289

19.469

30.734

0.99903

False

8

18.742

18.545

19.168

19.492

30.876

7.9921

False

16

19.458

18.736

19.801

31.829

34.98

15.983

False

32

26.891

31.555

35.423

35.911

36.976

31.948

False

64

30.669

34.484

37.03

37.69

51.036

63.884

False

128

36.049

37.917

41.027

41.658

56.785

127.71

False

256

44.443

37.834

47.662

68.798

246.81

255.26

False

512

44.948

43.888

58.187

60.367

92.103

509.62

False

1024

62.029

56.187

77.908

103.34

256.1

1016.4
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

9.3198

9.3654

10.023

10.188

12.775

0.99957

False

8

12.106

9.3858

10.233

10.594

55.442

7.9964

False

16

9.8635

9.6687

10.539

11.214

16.649

15.992

False

32

12.949

13.469

17.03

17.396

20.391

31.974

False

64

16.283

16.661

22.428

23.305

29.454

63.916
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

10.442

10.403

11.143

11.288

14.557

0.9995

False

64

17.824

19.476

21.854

22.548

29.683

63.926

False

128

22.386

23.455

26.247

26.803

38.527

127.78

False

512

41.412

33.274

51.051

104.41

256.69

509.64

False

768

41.456

41.087

61.155

69.104

109.63

763.13

False

1024

53.31

45.379

78.945

126.14

264.88

1015.7

False

1200

52.625

49.018

76.847

88.914

197.31

1188.5

False

1400

76.331

72.281

112.11

127.39

257.04

1383.3

False

1536

420.58

434.14

717.81

798.91

962.2

1481.7
Chunk size (ms): 160
Batch size: 512
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

18.692

18.507

19.241

19.411

30.78

0.99903

False

8

19.86

18.365

19.115

19.374

38.523

7.992

False

16

19.811

18.676

19.781

33.15

34.86

15.976

False

32

27.006

31.455

35.011

35.411

37.316

31.948

False

64

30.816

34.464

36.462

37.11

49.889

63.884

False

128

35.843

37.663

40.239

40.85

58.437

127.71

False

256

37.217

37.52

44.8

46.366

63.29

255.3

False

1024

65.116

58.766

77.708

144.44

251.59

1016.6
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 190

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

11.518

10.501

11.753

12.329

29.332

0.99948

8

13.042

12.727

14.303

16.54

27.45

7.9934

16

17.579

16.357

25.071

26.493

42.529

15.974

32

21.415

18.903

27.705

28.62

65.338

31.924

48

32.285

32.166

34.611

35.804

102.55

47.839

64

33.933

36.076

39.682

41.26

120.46

63.75
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 900

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

14.345

12.899

18.496

21.621

49.489

0.99941

64

43.724

41.908

48.17

138.95

140.61

63.715

128

76.158

69.027

80.239

198.79

277.37

126.88

256

113.72

89.307

128.73

294.93

488.96

251.9

384

150.8

133.5

170.69

465.93

722.34

374.76

512

198.83

173.53

280.75

577.5

975.18

495.82
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

365.61

False

32

3638

True

32

101.5
Chunk size (ms): 320
Language model: n-gram
Maximum effective # of streams with n-gram language model: 270

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

13.377

12.014

14.8

15.721

17.616

0.99944

8

20.362

19.784

23.525

26.147

33.041

7.9919

16

28.97

28.08

34.588

37.939

52.757

15.97

32

42.96

38.11

55.592

57.928

94.558

31.904

48

58.84

67.281

75.958

77.311

136.62

47.794

64

79.065

88.762

99.511

109.32

181.85

63.634
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1240

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

17.358

16.19

19.135

23.812

52.552

0.99925

64

86.236

102.67

110.26

112.32

119.81

63.754

128

204.03

205.92

220.27

223.14

250.96

126.93

256

315.08

321.68

395.18

408.04

502.56

251.93

384

423.9

421.51

577.25

664

857.63

373.82

512

573

563.93

874.58

1039.3

1263.3

492.08
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

335.32

False

32

2876.4
Chunk size (ms): 160
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

21.6

21.1

22.8

23.4

32.5

1.0

8

31.5

31.2

33.8

35.2

51.5

8.0

16

45.5

45.4

47.8

53.5

79.2

16.0

32

67.4

61.8

89.0

90.6

119.4

31.8

48

98.9

116.6

127.1

134.2

182.2

47.6
Chunk size (ms): 960
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

21.1

20.3

23.0

24.7

49.2

1.0

64

161.7

197.5

204.8

208.3

212.7

63.6

128

369.0

396.0

432.2

450.6

455.8

126.2
Language model: n-gram

# of streams

Throughput (RTFX)

1

264.5

32

882.6
Chunk size (ms): 160
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

29.2

28.8

30.4

31.0

51.3

1.0

True

1

42.0

29.2

42.7

44.9

432.5

1.0

False

8

38.9

38.4

40.9

41.5

72.6

8.0

True

8

45.0

39.6

58.2

60.3

119.8

8.0

False

16

55.1

53.9

56.9

69.1

114.0

15.9

True

16

68.7

54.3

93.4

117.3

235.3

15.9

False

32

85.0

68.0

111.5

114.5

158.0

31.8

True

32

115.1

108.5

188.4

205.2

376.7

31.6

False

48

127.5

148.5

158.0

191.3

265.8

47.6

True

48

213.6

204.7

283.7

343.9

555.8

47.3
Chunk size (ms): 960
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

59.8

59.2

61.8

63.2

92.3

1.0

True

1

85.1

72.2

76.2

83.5

514.8

1.0

False

64

255.4

304.4

310.3

313.2

315.0

63.4

True

64

372.3

422.1

463.4

469.2

471.6

63.1

False

128

478.7

513.0

528.6

666.1

686.7

125.3

True

128

687.7

695.9

776.2

1113.9

1620.6

123.7
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

20.7

False

32

336.4

True

32

260.5
Chunk size (ms): 160
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

19.671

20.104

21.685

21.96

38.539

0.99884

8

31.194

31.719

35.482

36.195

66.154

7.9835

16

45.007

46.339

50.019

51.456

92.321

15.953

32

61.018

56.473

77.184

79.764

136.4

31.801

48

79.726

87.697

98.868

100.64

172.36

47.647

64

102.58

117.05

125.58

130.69

271.66

63.453
Chunk size (ms): 960
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

24.776

24.027

30.257

32.001

63.361

0.99901

64

114.75

133.68

149.83

153.73

157.22

63.679

128

235.08

244.42

285.45

290.93

374.07

126.5

256

367.18

365.85

468.17

506.79

691.56

250.49

384

485.58

465.62

668.72

772.08

999.89

371.5

512

637.89

635.49

970.62

1132.4

1399.5

489.51
Language model: n-gram

# of streams

Throughput (RTFX)

1

180.37

32

1037.8
Chunk size (ms): 160
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

14.202

13.052

17.88

19.466

25.956

0.99947

8

24.437

22.858

29.938

38.885

49.618

7.9913

16

36.536

34.467

44.059

54.708

81.838

15.973

32

48.218

52.974

64.388

81.382

99.757

31.904

48

69.337

78.074

89.509

110.77

143.67

47.784

64

92.235

101.77

114.86

181.18

204.97

63.62
Chunk size (ms): 960
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

19.475

17.506

22.395

23.533

78.759

0.99942

64

120.23

118.25

207.22

215.26

240.07

63.754

128

246.07

227.94

373.71

448.51

456.11

126.96

256

426

422.2

724.17

801.8

895.13

251.23

384

572.76

545.31

976.51

1096.7

1214.7

372.22

512

781.89

704.9

1342

1422.6

1613.6

491.52
Language model: n-gram

# of streams

Throughput (RTFX)

1

349.89

32

2885.7
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

20.0

18.8

20.0

20.8

36.8

1.0

8

21.5

21.1

22.5

23.6

41.5

8.0

16

24.5

23.7

25.3

35.4

53.5

16.0

32

32.9

35.6

41.5

42.8

78.7

31.9

48

39.3

44.2

47.8

49.7

99.4

47.8

64

46.9

51.4

55.0

56.4

152.1

63.7
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

23.3

21.6

26.8

29.7

71.9

1.0

64

60.1

59.4

68.0

152.8

157.1

63.7

128

89.0

91.7

101.0

196.8

289.9

126.8

256

133.7

108.4

173.2

383.8

552.3

251.6

384

183.2

150.9

251.1

558.0

819.9

374.2

512

256.3

205.5

484.9

665.2

1351.4

494.5
Language model: n-gram

# of streams

Throughput (RTFX)

1

286.7

32

2490.4
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

21.8

21.2

22.1

22.4

39.6

1.0

True

1

25.6

22.5

35.0

35.5

71.3

1.0

False

8

27.5

26.9

28.2

29.7

61.9

8.0

True

8

34.7

28.2

48.4

50.4

120.4

8.0

False

16

36.9

35.1

36.6

57.6

97.4

15.9

True

16

55.2

56.0

82.7

84.6

193.7

15.9

False

32

51.6

39.9

65.7

68.2

131.3

31.8

True

32

71.6

64.6

146.0

150.3

303.5

31.7

False

48

68.0

76.7

85.8

92.2

168.5

47.7

True

48

101.9

83.0

178.4

189.3

479.8

47.3
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

55.8

55.5

56.3

61.4

61.8

1.0

True

1

70.0

69.2

70.6

79.2

80.0

1.0

False

64

175.2

194.2

197.8

273.0

312.9

63.5

True

64

263.9

292.5

313.6

458.7

465.9

63.1

False

128

252.2

262.2

273.2

397.9

471.7

126.1

False

256

438.2

419.6

492.1

819.3

1027.5

248.0

False

384

759.5

626.9

1600.5

1968.1

2799.9

364.6

False

512

2054.4

1823.7

3943.4

4720.0

5667.5

456.1

False

512

2015.4

1795.2

3924.8

4561.5

5507.2

457.1
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

21.9

True

1

19.9

False

32

420.4

True

32

308.5

Speaker Diarization

# of streams

Throughput (RTFX)

Average Latency (ms)

False

1

158.3

352.24

False

32

1631.3

1018.84
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 280

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

11

10.3

11.2

12.4

30

1

8

20

19

26

30

42

7.99

16

28

26

35

40

56

15.97

32

35

35

48

52

73

31.9

64

50

55

66

70

100

63.8
Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1180

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

14

11.5

20

30

60

1

64

70

70

90

100

170

63.8

128

88

84

110

190

250

127.4

256

128

117

164

300

460

254.4
Language model: n-gram

# of streams

Throughput (RTFX)

32

440
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

70

False

32

193.5
Language model: none

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

6.2

False

32

43.3
Chunk size (ms): 320
Language model: n-gram

# of streams

Throughput (RTFX)

Average Latency (ms)

1

1.0

86.53

8

7.9

136.34

16

15.8

163.55

32

31.4

253.70

48

44.8

991.17

64

58.9

1180.73
Chunk size (ms): 1600
Language model: n-gram

# of streams

Throughput (RTFX)

Average Latency (ms)

1

1.0

87.29

64

63.0

433.03

128

125.0

586.62

256

246.3

836.96

384

337.1

2274.47

512

342.7

7912.27
Language model: n-gram

# of streams

Throughput (RTFX)

Average Latency (ms)

1

85.2

642.87

32

1056.6

1606.57
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

6.8779

6.8674

7.4391

7.5684

9.3649

0.99969

False

8

6.8977

6.8113

7.4702

7.619

10.675

7.9973

False

16

7.0845

6.8922

7.693

10.252

12.19

15.993

False

32

8.1702

8.189

9.4168

9.7376

11.056

31.979

False

64

9.8601

10.209

12.944

14.481

19.038

63.938
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

7.2996

7.2962

7.9478

8.0403

9.6036

0.99967

False

64

11.345

12.155

13.826

14.269

16.364

63.956

False

128

13.342

13.902

16.947

17.545

22.028

127.85

False

512

20.225

18.947

27.662

29.147

56.778

510.2

False

768

25.706

25.211

36.597

38.724

76.253

764.53

False

1024

29.14

27.975

42.388

44.576

90.481

1017.9

False

1200

32.463

29.9

46.588

51.909

105.95

1191.8

False

1400

37.173

34.784

53.073

56.855

141.25

1388.9

False

1536

36.885

32.379

53.545

62.765

157.77

1522.9

False

1600

36.087

30.177

49.912

59.17

179.99

1586

False

1800

42.966

39.187

59.733

75.261

191.81

1782.1

False

1900

49.173

43.37

67.431

92.37

219.03

1879.4

False

2048

68.045

51.3

146.13

158.86

261.37

2017.1
Chunk size (ms): 160
Batch size: 512
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

12.444

12.376

12.906

13.013

18.043

0.99942

False

8

12.741

12.438

13.14

13.514

22.416

7.9952

False

16

12.915

12.532

13.234

13.619

22.85

15.99

False

32

17.361

15.038

22.765

23.092

23.972

31.968

False

64

19.901

21.783

23.424

23.837

29.61

63.927

False

128

21.303

21.976

25.136

25.751

31.391

127.82

False

256

24.512

24.085

30.055

31.181

39.61

255.45

False

512

30.134

28.604

38.903

42.854

67.986

510.28
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

6.73

6.7275

7.24

7.38

9.1549

0.9997

False

8

6.8815

6.8872

7.4292

7.5724

9.3705

7.9972

False

16

6.9779

6.8771

7.54

7.8017

11.847

15.994

False

32

9.0406

9.5888

11.575

11.867

13.332

31.984

False

64

10.799

11.078

14.43

14.93

20.552

63.946
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

7.1228

7.0957

7.6056

7.7923

8.8923

0.99969

False

64

11.336

12.1

13.813

14.233

16.67

63.953

False

128

13.948

14.448

16.885

17.413

22.096

127.87

False

512

19.026

17.709

26.755

28.675

49.664

510.3

False

768

24.066

23.871

31.868

33.371

69.591

764.6

False

1024

30.974

28.537

42.227

47.803

119.99

1017.9

False

1200

33.547

31.617

47.516

50.455

105.13

1192

False

1400

37.377

36.564

51.559

54.257

130.66

1389.3

False

1536

38.119

36.099

51.648

59.066

155.08

1522.6

False

1600

35.435

29.044

55.451

65.966

161.09

1585.9

False

1800

46.363

41.949

63.496

77.715

201.08

1781.9

False

1900

47.081

41.081

64.441

106.81

227.69

1879.2

False

2048

61.504

54.001

85.425

136.2

237.21

2022.6
Chunk size (ms): 160
Batch size: 512
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

12.661

12.575

13.036

13.191

18.433

0.99942

False

8

12.754

12.622

13.156

13.278

19.236

7.995

False

16

12.995

12.713

13.314

13.527

22.937

15.989

False

32

17.463

13.398

23.166

23.425

24.563

31.967

False

64

20.303

22.382

23.879

24.234

30.662

63.926

False

128

21.795

22.476

25.45

25.983

32.687

127.82

False

256

25.01

25.387

29.722

30.514

40.305

255.5

False

512

28.813

28.183

35.348

38.845

62.89

510.31
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

12.371

12.296

13.228

13.433

16.529

0.99946

False

8

12.148

11.959

12.796

13.255

20.194

7.9953

False

16

14.212

12.444

20.553

21.184

22.047

15.986

False

32

16.846

18.054

22.086

22.783

24.608

31.969

False

64

22.53

22.629

30.855

31.844

38.057

63.894
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

12.958

12.95

13.712

14.12

18.037

0.99942

False

64

23.416

25.294

28.251

28.882

35.702

63.911

False

128

29.015

30.204

35.068

36.297

49.566

127.73

False

512

51.161

49.44

72.242

77.665

122.78

508.46

False

768

65.477

62.23

96.15

107.56

198.81

760.42

False

1024

1517.4

1397.8

2780.4

3069.3

3296.7

921.24
Chunk size (ms): 160
Batch size: 512
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

28.669

28.317

29.369

29.812

49.157

0.99843

False

128

56.415

58.432

63.042

64.204

87.99

127.58

False

512

83.634

84.935

108.17

111.73

147.31

508.13
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

12.561

12.545

13.416

13.59

16.564

0.99946

False

8

12.619

12.282

13.387

17.356

21.299

7.9931

False

16

13.696

12.469

20.405

21.353

22.726

15.989

False

32

18.252

20.028

22.548

23.277

27.211

31.967

False

64

21.884

22.712

30.315

31.805

41.011

63.889
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

13.105

12.976

13.926

14.423

18.253

0.99936

False

64

22.395

24.581

27.628

28.581

37.322

63.909

False

128

28.884

30.214

34.613

35.91

49.18

127.72

False

512

51.879

50.741

72.349

79.002

131.65

508.49

False

768

69.25

67.633

103.1

116.91

197.84

760.82

False

1024

1225

1141

2234.3

2432.4

2564.6

940.66
Chunk size (ms): 160
Batch size: 512
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

28.612

28.288

29.43

29.858

48.205

0.99845

False

128

55.906

57.788

61.716

62.962

84.515

127.6

False

512

89.17

89.938

104.61

107.36

147.78

508.37

False

1024

132.14

128.32

162.67

178.09

274.04

1009.3
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

21.4

21.0

22.2

22.6

35.0

1.0

8

28.1

27.0

28.5

33.3

63.3

8.0

16

39.8

38.8

50.0

52.0

100.0

15.9

32

60.2

68.4

74.7

76.4

182.3

31.8

48

87.7

97.8

103.9

125.2

278.2

47.6

64

133.6

130.1

150.9

311.0

616.6

63.3
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

22.8

21.7

26.7

30.4

48.7

1.0

64

143.5

155.1

166.9

303.0

355.6

63.3

128

246.4

206.7

309.9

479.8

606.6

125.6

256

514.9

421.4

1014.2

1355.0

1910.0

245.0

384

1672.4

960.7

4440.0

5127.3

5731.1

355.7

512

6575.1

6269.7

12263.0

13473.0

17069.0

367.8
Language model: n-gram

# of streams

Throughput (RTFX)

1

224.3

32

994.4
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

50.8

28.9

39.4

160.6

583.1

1.0

8

39.0

35.1

48.2

51.2

135.1

7.9

16

55.6

48.7

68.8

72.2

239.2

15.7

32

79.6

71.0

95.7

127.1

347.0

30.9

48

114.8

116.5

170.8

188.0

482.4

45.5

64

177.8

172.6

234.3

247.5

709.6

59.5

128

4827.4

4660.0

8580.3

9186.2

10417.0

64.8
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

71.6

50.9

54.1

313.6

313.6

1.0

64

236.3

244.1

309.8

521.3

522.9

61.0

128

446.8

425.9

604.8

1075.0

1332.1

114.0
Language model: n-gram

# of streams

Throughput (RTFX)

1

34.0

32

249.0

64

240.6

128

218.4
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

8.2446

8.2737

8.9387

9.1001

10.596

0.99962

False

8

9.8851

9.2595

12.281

12.637

13.505

7.9959

False

16

10.209

9.8703

12.803

13.367

14.729

15.989

False

32

10.9

10.859

13.01

13.594

16.48

31.973

False

64

11.578

11.34

14.049

15.159

20.549

63.915
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

8.411

8.4068

9.2147

9.3237

10.514

0.99961

False

64

12.585

12.559

15.176

16.075

19.094

63.928

False

128

13.413

13.325

15.934

16.922

23.59

127.78

False

512

14.238

13.678

17.866

19.483

31.924

509.9

False

768

12.037

11.673

14.363

15.305

25.22

763.76

False

1024

13.714

12.708

17.339

19.894

36.403

1017.2

False

1200

18.224

14.87

29.025

32.151

50.36

1189.9

False

1400

20.6

17.299

32.293

35.897

64.201

1387.6

False

1536

23.859

18.99

34.541

40.575

118.28

1519.8

False

1600

30.51

28.151

47.819

53.31

100.97

1581.4

False

1800

35.293

32.998

53.475

60.241

124.37

1775.1

False

1900

38.854

35.999

57.567

65.481

135.38

1872.2

False

2048

45.07

41.298

64.863

78.917

167.69

2018
Chunk size (ms): 160
Batch size: 512
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

10.249

10.296

11.033

11.196

13.066

0.99953

False

128

16.017

16.112

18.956

19.859

26.497

127.78

False

512

17.351

16.902

20.635

22.105

32.986

510.14

False

1024

18.195

16.773

21.314

23.303

55.669

1017.8

False

1536

26.048

21.309

35.877

40.74

139.69

1520.9

False

2048

38.29

32.216

49.114

57.507

240.7

2023.3

False

2304

42.732

34.626

51.169

68.542

294.49

2270.1

False

2560

50.887

39.631

58.836

110.92

389.22

2514.3

False

2660

57.744

40.382

62.382

174.63

462.1

2613.2

False

2816

69.095

47.113

94.185

236.39

500.54

2765.5

False

2900

67.415

46.68

69.156

234.41

498.27

2848.3

False

3072

75.442

53.944

106.45

248.52

487.37

3012.1

False

3200

125.18

79.743

191.59

393.55

882.26

3124.3
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

8.1796

8.1787

8.8166

9.0281

10.719

0.99961

False

8

9.3078

8.6797

12.207

12.649

13.474

7.9952

False

16

10.134

10.024

12.218

12.941

14.633

15.989

False

32

10.744

10.83

12.723

13.236

15.016

31.975

False

64

11.502

11.444

13.475

14.079

22.677

63.924
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

8.5372

8.5197

9.2041

9.4366

11.359

0.99959

False

64

11.978

11.923

14.126

14.808

18.619

63.931

False

128

12.431

12.186

14.926

15.859

21.957

127.81

False

512

12.548

12.311

14.845

15.707

23.076

510.3

False

768

11.92

11.621

14.11

15.069

23.01

763.85

False

1024

13.462

12.48

18.01

20.191

30.469

1017.2

False

1200

16.255

13.846

24.125

27.751

39.67

1190.6

False

1400

23.118

20.698

35.884

39.945

75.565

1386

False

1536

26.108

24.099

39.843

46.296

86.367

1519.6

False

1600

26.098

23.279

41.268

47.212

74.552

1582.1

False

1800

29.678

25.275

43.54

54.79

142.12

1773.8

False

1900

32.27

28.398

47.134

56.883

139.23

1874

False

2048

42.459

32.994

54.56

109.77

261.38

2017.1
Chunk size (ms): 160
Batch size: 512
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

10.103

10.074

10.685

10.81

12.77

0.99954

False

128

13.974

13.896

16.666

17.375

22.06

127.75

False

512

17.303

16.757

20.149

21.476

38.789

510.33

False

1024

16.876

16.165

20.444

22.711

35.715

1017.5

False

1536

24.207

20.32

31.206

35.038

127.42

1521.7

False

2048

36.159

32.226

47.931

54.457

209.88

2021.9

False

2304

41.127

34.722

50.98

66.062

279.1

2269.7

False

2560

50.025

37.925

56.889

121.89

374.33

2519.1

False

2660

53.262

41.457

58.476

135.9

378.46

2610

False

2816

53.01

39.894

57.493

127.29

381.73

2765.2

False

2900

61.812

42.527

64.545

213.21

462.74

2849.1

False

3072

66.485

44.56

89.688

234.09

468.94

3006.7

False

3200

71.739

49.871

119.44

222.32

457.01

3117.9

False

3328

79.213

49.79

131.01

291.2

521.08

3253.7

False

3344

86.695

55.214

164.71

308.94

544.63

3266.8

False

3360

93.402

52.248

191.35

352.66

604.33

3280.4

False

3376

90.173

57.055

164.24

315.45

559.13

3299.8

False

3392

88.125

55.751

165.1

311.69

537.1

3314.4

False

3400

95.145

54.971

200.94

344

570.42

3324.7

False

3440

93.521

54.909

186.43

326.46

547.29

3357.5

False

3480

114.77

81.739

225.3

330.27

563.16

3395.4

False

3520

101.54

64.729

190.79

334.05

572.13

3435.9

False

3560

134.89

105.14

214.71

355.01

624.83

3462.8
Chunk size (ms): 320
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

1.0

16

15.6

32

31.1

64

61.5

128

119.1
Chunk size (ms): 1600
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

1.0

64

62.3

128

122.8
Language model: n-gram

# of streams

Throughput (RTFX)

1

27.3

32

743.5

32

876.3

64

1008.0

128

1463.5
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

15.6

14.9

16.5

17.2

21.1

1.0

8

19.0

16.9

24.2

25.0

39.4

8.0

16

25.8

25.2

31.9

34.8

81.3

15.9

32

28.5

26.6

36.9

42.6

94.3

31.9

48

29.9

27.5

39.0

47.3

109.5

47.8

64

33.6

30.6

43.4

56.0

138.4

63.7
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

18.6

16.8

23.9

26.7

58.4

1.0

64

54.4

34.7

96.2

192.8

231.3

63.5

128

63.5

35.5

127.7

226.8

411.3

126.2

256

104.6

46.7

243.5

442.4

820.6

249.2

384

156.9

46.2

482.4

727.2

1339.6

369.5

512

227.1

48.5

847.0

1340.9

1724.3

486.0
Language model: n-gram

# of streams

Throughput (RTFX)

1

359.9

32

1986.6
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

39.3

19.9

25.9

106.7

554.0

1.0

8

23.6

21.5

28.6

29.7

68.6

7.9

16

31.5

31.9

46.9

48.1

76.1

15.8

32

39.2

37.7

54.5

63.4

147.7

31.5

48

47.6

43.6

65.6

71.0

205.2

47.1

64

52.5

47.1

72.9

82.0

227.8

62.6

128

67.6

57.9

94.0

117.4

334.2

122.3
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

68.1

32.7

41.0

447.6

447.6

1.0

64

144.5

137.8

168.4

336.1

345.7

62.0

128

161.2

138.6

231.1

382.9

596.0

121.1
Language model: n-gram

# of streams

Throughput (RTFX)

1

53.4

32

765.8

64

970.1

128

1198.5
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

8.8432

8.8477

9.5505

9.6596

11.742

0.9996

False

8

10.269

9.5628

13.421

14.212

15.327

7.9949

False

16

10.802

10.494

13.224

14

17.13

15.987

False

32

12.073

12.093

14.491

15.195

18.271

31.964

False

64

13.088

13

15.296

16.148

24.849

63.891
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

9.1023

9.1922

9.8435

10.035

11.977

0.99958

False

64

13.062

13.083

15.439

16.211

20.175

63.901

False

128

13.428

13.3

16.041

16.94

21.905

127.69

False

512

13.425

13.031

15.644

16.653

27.913

508.18

False

768

13.971

13.434

16.435

17.643

30.498

759.96

False

1024

16.482

15.411

19.775

21.759

49.261

1009.8

False

1200

18.914

16.873

24.318

27.284

74.301

1180.2

False

1400

21.881

18.582

32.064

35.742

60.961

1373.1

False

1536

26.82

23.753

39.765

44.387

89.55

1503.8

False

1600

28.996

26.701

41.83

46.927

91.28

1563.1

False

1800

42.653

33.548

53.016

97.952

289.7

1756.1

False

1900

46.675

42.718

65.038

78.312

166.31

1851.6

False

2048

61.418

52.181

83.959

136.97

310.67

1989.1
Chunk size (ms): 160
Batch size: 512
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

10.589

10.589

11.276

11.397

13.584

0.99953

False

128

15.434

15.243

18.145

19.159

28.775

127.68

False

512

16.592

16.21

19.179

20.214

32.016

508.3

False

1024

19.314

18.063

23.075

25.556

51.958

1010.5

False

1536

31.993

27.713

44.31

50.687

158.37

1504.9

False

2048

55.187

40.452

63.492

138.32

425.84

1990.9

False

2304

63.352

45.59

74.026

173.93

477.78

2241.1

False

2560

66.609

49.722

78.211

167.03

484.03

2479.9

False

2660

85.475

59.404

121.76

292.75

538.23

2573

False

2816

189.08

68.859

530.18

625.52

737.76

2713.3

False

2900

114.44

69.66

240.97

409.27

629.43

2796.5
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

8.8064

8.7938

9.4851

9.5911

12.234

0.99959

False

8

10.583

9.888

13.273

13.731

15.083

7.995

False

16

11.334

11.3

13.548

14.074

17.05

15.987

False

32

11.899

11.915

14.109

14.789

18.165

31.967

False

64

12.544

12.407

14.681

15.396

24.435

63.901
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

8.9142

8.8878

9.6569

9.8115

11.036

0.99956

False

64

13.14

13.1

15.756

16.396

20.358

63.896

False

128

13.967

13.797

16.327

17.188

24.581

127.69

False

512

13.207

12.915

15.535

16.472

24.926

508.04

False

768

14.056

13.467

16.274

17.434

35.754

759.81

False

1024

15.618

15.064

18.421

19.801

39.124

1009.6

False

1200

17.784

16.094

22.843

25.467

53.504

1179.8

False

1400

20.007

17.772

26.287

29.875

58.688

1373.1

False

1536

23.1

20.383

31.862

37.607

71.958

1504.3

False

1600

25.42

22.79

36.264

39.967

83.48

1565.6

False

1800

34.694

30.556

46.971

67.956

181.13

1755.9

False

1900

39.872

33.184

52.56

78.21

248.34

1852.2

False

2048

44.083

37.237

56.82

98.196

205.05

1991.9
Chunk size (ms): 160
Batch size: 512
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

10.492

10.435

11.164

11.333

13.549

0.99954

False

128

15.628

15.477

18.397

19.43

25.624

127.66

False

512

15.9

15.54

18.395

19.443

30.212

508.07

False

1024

19.744

18.876

24.105

26.162

46.466

1009.9

False

1536

28.198

24.129

36.356

40.756

117.38

1505.4

False

2048

40.386

33.187

49.646

65.298

272.73

1993.4

False

2304

53.327

38.945

58.781

131.24

397.79

2234.8

False

2560

73.473

45.272

95.892

314.51

558.6

2477.8

False

2660

82.113

50.856

119.19

351.3

620.47

2572.3

False

2816

165.37

58.744

276.52

609.57

2401.3

2707.1
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

6.5

6.5177

7.1986

7.289

8.4601

0.99971

False

8

6.5865

6.5502

7.2677

7.5032

9.4298

7.9975

False

16

6.6912

6.5401

7.3835

8.9426

10.891

15.994

False

32

7.9023

7.1683

10.47

10.777

11.627

31.985

False

64

9.4765

9.7151

12.582

13.098

16.526

63.951
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

7.2213

7.2542

8.0754

8.2167

9.1137

0.9997

False

64

11.101

11.66

13.499

14.014

16.656

63.952

False

128

12.843

12.985

16.024

16.794

20.209

127.86

False

512

17.014

16.041

23.246

26.099

47.203

510.43

False

768

20.807

19.452

28.226

30.497

57.569

764.85

False

1024

22.971

21.991

31.266

35.343

83.075

1018.7

False

1200

19.078

16.5

27.417

34.5

75.771

1192.2

False

1400

25.962

23.282

36.851

44.554

113.23

1390.1

False

1536

23.892

18.952

37.154

43.116

119.01

1522.9

False

1600

26.048

20.214

38.821

50.64

166.21

1587.4

False

1800

28.166

23.225

37.577

52.355

173.41

1784.4

False

1900

33.337

24.916

45.29

87.372

259.61

1881.9

False

2048

37.408

29.492

51.469

92.054

238.55

2026
Chunk size (ms): 160
Batch size: 512
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

10.72

10.69

11.398

11.47

14.184

0.99953

False

128

18.073

18.475

21.394

22.286

27.606

127.83

False

512

21.36

21.18

26.099

27.513

44.999

510.45

False

1024

25.087

23.687

31.35

34.922

67.471

1018.5

False

1536

26.699

22.846

33.621

40.603

122.32

1522.7

False

2048

32.872

24.931

37.148

66.707

244.65

2029.5

False

2304

40.481

29.531

49.248

89.495

335.95

2278.5

False

2560

49.618

31.483

55.026

170.05

446.53

2521.1

False

2660

62.069

36.802

64.354

208.28

649.52

2622.3

False

2816

64.267

39.232

108.28

219.76

535.4

2775.7

False

2900

62.966

41.514

88.485

190.05

536.05

2854

False

3072

70.539

43.598

106.74

217.8

615

3024.8

False

3200

75.827

45.38

134.58

211.97

628.82

3143.7

False

3328

83.858

49.709

128.45

243.92

759.34

3269.2

False

3344

91.564

49.926

149.26

275.66

914.17

3280.9

False

3360

99.296

53.235

165.73

322.31

915.14

3297.9

False

3376

89.859

52.48

158.29

251.62

700.27

3310.7

False

3392

102.74

56.563

190.97

297.06

801.46

3332.3

False

3400

102.95

59.93

194.41

284.49

713.37

3337

False

3440

99.389

63.103

162.23

261.93

730.9

3375.3

False

3480

137.84

101.16

210.09

342.77

977.4

3409.2

False

3520

156.89

121.19

278.03

373.16

790.52

3435.5

False

3560

159.77

134.09

268.13

366.23

781.01

3478.2
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

6.6701

6.6489

7.354

7.4518

8.828

0.99969

False

8

6.5791

6.3351

8.1479

9.7835

10.665

7.9975

False

16

6.8322

6.559

7.6959

10.095

10.816

15.995

False

32

7.8434

7.1696

10.202

10.557

11.701

31.985

False

64

9.1657

9.3726

11.965

12.508

16.393

63.951
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

7.3929

7.4194

7.9899

8.1147

9.5374

0.99968

False

64

10.216

9.8267

12.914

13.695

17.955

63.947

False

128

12.371

12.849

15.545

16.089

21.189

127.84

False

512

15.549

14.736

20.316

24.648

39.916

510.41

False

768

16.53

15.395

22.318

25.153

51.433

764.67

False

1024

19.287

16.559

27.875

32.221

79.574

1018.7

False

1200

22.328

21.184

31.077

34.482

80.284

1192.9

False

1400

19.41

16.018

22.828

27.11

112.14

1390.2

False

1536

22.152

16.979

34.755

42.675

119.44

1523.2

False

1600

23.487

18.455

33.703

40.138

129.02

1587.1

False

1800

25.131

18.319

37.198

52.253

160.77

1784.3

False

1900

30.163

22.676

39.684

78.377

235.29

1882.7

False

2048

34.584

25.361

45.718

94.462

253.53

2026.6
Chunk size (ms): 160
Batch size: 512
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

10.911

10.884

11.559

11.746

14.808

0.9995

False

128

17.71

17.947

21.032

21.723

26.792

127.83

False

512

22.297

21.848

26.887

28.306

47.115

510.5

False

1024

21.01

20.164

24.59

26.304

54.452

1019.2

False

1536

24.731

21.563

28.996

35.824

115.69

1524.4

False

2048

30.275

23.1

34.012

60.487

228.36

2028.1

False

2304

38.223

25.894

42.475

94.608

346.88

2278.5

False

2560

43.504

31.624

48.372

119.45

360.19

2522.5

False

2660

47.582

32.608

57.948

143.91

400.52

2623.3

False

2816

55.275

36.443

75.247

175.75

477.57

2774.5

False

2900

59.572

37.497

94.487

188.59

490.18

2855.6

False

3072

64.6

40.668

103.5

183.69

544.85

3022.5

False

3200

80.81

41.548

114.59

281.22

904.97

3144.9

False

3328

77.025

43.504

134.75

217.9

631.96

3268.6

False

3344

86.072

46.24

145.43

262.85

718.7

3278.2

False

3360

85.099

52.004

152.42

224.09

671

3295.2

False

3376

86.414

45.728

170.18

241.21

637.53

3306.7

False

3392

92.087

46.623

157.82

304.39

806.74

3330.1

False

3400

120.93

51.78

187.59

441.18

1260.8

3328.2

False

3440

100.14

49.988

187.14

302.56

837.4

3377.1

False

3480

107.72

52.309

209.01

281.07

752.3

3415.7

False

3520

100.94

51.037

181.58

275.66

739.68

3455.1

False

3560

111.13

63.295

194.12

275.58

803.16

3490.4

False

3584

126.55

95.373

202.5

298.79

883.77

3512.9

On-Prem Hardware Specifications#

GPU

NVIDIA DGX A100 40GB

CPU

Model

AMD EPYC 7742 64-Core Processor

Thread(s) per core

2

Socket(s)

2

Core(s) per socket

64

NUMA node(s)

8

Frequency boost

enabled

CPU max MHz

2250

CPU min MHz

1500

RAM

Model

Micron DDR4 36ASF8G72PZ-3G2B2 3200MHz

Configured Memory Speed

2933 MT/s

RAM Size

32x64GB (2048GB Total)

GPU

NVIDIA H100 80GB HBM3

CPU

Model

Intel(R) Xeon(R) Platinum 8480CL

Thread(s) per core

2

Socket(s)

2

Core(s) per socket

56

NUMA node(s)

2

CPU max MHz

3800

CPU min MHz

800

RAM

Model

Micron DDR5 MTC40F2046S1RC48BA1 4800MHz

Configured Memory Speed

4400 MT/s

RAM Size

32x64GB (2048GB Total)

GPU

NVIDIA L40S

CPU

Model

AMD EPYC 7763 64-Core Processor

Thread(s) per core

1

Socket(s)

2

Core(s) per socket

64

NUMA node(s)

8

Frequency boost

enabled

CPU max MHz

3529

CPU min MHz

1500

RAM

Model

Samsung DDR4 M393A4K40DB3-CWE 3200MHz

Configured Memory Speed

3200 MT/s

RAM Size

16x32GB (512GB Total)

Model Accuracy#

ASR models are evaluated with the following metrics. Lower values indicate better accuracy, with 0% representing perfect transcription.

  • Word Error Rate (WER): Used for word-based languages (English, Spanish, and French). Measures the minimum word substitutions, insertions, and deletions needed to match the reference transcript, divided by total reference words.

  • Character Error Rate (CER): Used for character-based languages (Chinese, Japanese, and Mandarin). Measures the minimum character edits needed, divided by total reference characters.

  • Concatenated minimum-Permutation Word Error Rate (cpWER): Used for speaker diarization. Calculated by:

    1. Concatenating all utterances per speaker for both reference and hypothesis.

    2. Computing WER across all possible speaker permutations of the hypothesis.

    3. Selecting the lowest WER (best permutation).

Model Name

Language

Dataset

Best latency WER (%) ⬇️

Best throughput WER (%) ⬇️

Offline WER (%) ⬇️

Parakeet 1.1b CTC

en-US

MCV 7.1 test set

10.45

8.80

7.96

en-US

LibriSpeech test-other

6.34

4.74

4.09

en-US

CallHome (CH109)

46.09

41.35

39.61

en-US (Silero VAD)

LibriSpeech test-other

5.57

4.8

4.5

en-US (Telephony)

LibriSpeech test-other

7.33

5.11

4.17

en-US (Telephony)

CallHome (CH109)

30.13

27.82

28.91

en-US (Telephony) + Sortformer Diarizer

CallHome (CH109)

28.43 (cpWER)

-

-

Parakeet 0.6b TDT

en-US

AMI

-

-

11.46

en-US

Earnings22

-

-

11.65

en-US

Gigaspeech

-

-

9.15

en-US

LibriSpeech test-clean

-

-

2.01

en-US

LibriSpeech test-other

-

-

3.51

en-US

SPGISpeech

-

-

2.16

en-US

Tedlium

-

-

3.38

en-US

Voxpopuli

-

-

6.6

Parakeet 1.1b RNNT

en-US

MCV 7.1 test set

10.74

10.54

9.77

es-US

MLS test set

7.19

5.26

3.83

es-ES

Mediaspeech

16.15

14.42

11.51

fr-FR

MLS test set

11.41

9.10

6.36

de-DE

MLS test set

11.29

9.16

7.09

ru-RU

RuLS test set

21.44

19.23

17.39

Parakeet 0.6b CTC

en-US

MCV 7.1 test set

10.57

8.87

8.45

Parakeet 0.6b CTC

vi-VN

FLEURS Vietnamese test set

10

8.58

7.97

Parakeet 0.6b CTC

zh-CN

AISHELL1 & 2

5.81

5.84

6.09

Parakeet 0.6b CTC

es-US

MLS test set

9.14

6.15

5.34

Canary 1b

en-US

MCV 7.1 test set

Not supported

Not supported

6.78

es-US

MLS test set

Not supported

Not supported

3.54

de-DE

MLS test set

Not supported

Not supported

5.18

fr-FR

MLS test set

Not supported

Not supported

4.21

ru-RU

MCV 7.0 test set

Not supported

Not supported

10.33

es-ES

Mediaspeech

Not supported

Not supported

14.40

pt-BR

MCV 10.0 test set

Not supported

Not supported

5.83

Conformer 120m CTC

es-US

MCV 7.1 test set

6.75

6.26

5.66