Is this page helpful?

ASR NIM Performance#

This page provides latency and throughput benchmarks for the NVIDIA ASR NIM microservice in streaming and offline configurations across supported GPUs.

Evaluation Process#

The performance evaluation is done for the two different inference modes: streaming and offline.

Streaming Mode#

Streaming benchmarks use the riva_streaming_asr_client with the --simulate_realtime flag to simulate real-time microphone transcription. The client and server process audio chunks of the same duration. Each stream performs three iterations over a sample audio file (1272-135031-0000.wav) from the LibriSpeech dev-clean dataset. Refer to the Results section for chunk size values.

The source code for riva_streaming_asr_client is available at Riva C++ Clients.

The following command measures streaming performance:

riva_streaming_asr_client \
   --chunk_duration_ms=<chunk_duration> \
   --simulate_realtime=true \
   --automatic_punctuation=true \
   --num_parallel_requests=<num_streams> \
   --word_time_offsets=false \
   --print_transcripts=false \
   --interim_results=false \
   --num_iterations=<3*num_streams> \
   --audio_file=1272-135031-0000.wav \
   --output_filename=/tmp/output.json

The riva_streaming_asr_client returns three latency measurements:

  • intermediate latency: Latency of responses with is_final == false.

  • final latency: Latency of responses with is_final == true.

  • latency: Overall latency of all responses. This value is reported in the results tables.

The following diagram shows how these latencies are measured:

Schematic Diagram of Latencies Measured by Riva Streaming ASR Client

Offline Mode#

The following command measures maximum throughput in offline mode:

riva_asr_client \
   --automatic_punctuation=true \
   --num_parallel_requests=32 \
   --word_time_offsets=false \
   --print_transcripts=false \
   --num_iterations=96 \
   --audio_file=1272-135031-0000x5.wav \
   --output_filename=/tmp/output.json

The file 1272-135031-0000x5.wav is the 1272-135031-0000.wav audio file concatenated five times. The source code for riva_asr_client is available at Riva C++ Clients.

Results#

The following tables report latency and throughput for streaming and offline configurations. Throughput is measured in RTFX (duration of audio transcribed divided by computation time).

Note

All values are averages over three trials, rounded to the last significant digit based on standard deviation. If a standard deviation is less than 0.001 of the average, the value is rounded as if the standard deviation equals 0.001 of the average.

For the hardware used in these measurements, refer to the Hardware Specifications section.

Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 270

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

12.439

10.388

11.524

12.54

30.242

0.99949

8

13.006

12.508

14.673

17.203

29.539

7.9929

16

18.138

17.06

24.885

27.922

49.825

15.975

32

23.093

20.141

29.905

30.991

76.264

31.915

48

28.666

29.63

33.027

34.111

101.76

47.834

64

32.012

32.449

35.855

37.779

136.06

63.719
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1240

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

14.088

12.616

17.525

20.743

51.894

0.9994

64

42.41

37.123

43.153

157.71

163.47

63.68

128

61.41

49.455

61.86

197.73

307.18

126.82

256

93.439

67.938

98.617

315.2

558.71

251.39

384

123.79

93.576

124.35

472.36

848.59

373.93

512

166.85

117.95

318.19

615.9

1141.7

494.12
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

353.62

False

32

3707.4

True

32

170
Chunk size (ms): 320
Language model: n-gram
Maximum effective # of streams with n-gram language model: 270

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

16.946

14.233

17.459

18.954

21.387

0.99936

8

19.451

18.632

22.736

26.336

32.004

7.9924

16

24.811

23.88

28.443

32.162

42.753

15.978

32

33.166

30.537

43.692

47.19

68.485

31.929

48

44.522

49.317

57.667

60.352

93.513

47.855

64

55.794

60.906

71.632

74.103

117.78

63.755
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1240

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

19.719

19.153

22.404

25.33

53.192

0.99929

64

67.47

73.823

84.609

89.695

91.372

63.803

128

116.36

122.78

146.72

152.17

173.31

127.31

256

174.21

179.51

223.23

242.4

270.84

253.54

384

225.73

208.54

317.68

323.05

345.72

379.42

512

281.13

299.92

406.08

416.05

517.49

503.1
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

335.83

False

32

3890.1
Chunk size (ms): 160
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

27.7

26.0

27.7

28.5

42.8

1.0

8

33.1

32.9

35.1

35.9

54.1

8.0

16

43.4

42.7

45.7

57.2

73.9

16.0

32

59.7

48.4

78.1

80.1

105.4

31.9

48

99.1

106.5

110.5

112.2

187.5

47.7
Chunk size (ms): 960
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

23.7

22.9

26.3

27.6

50.4

1.0

64

135.7

160.3

167.9

171.1

174.8

63.7

128

273.6

300.2

314.6

319.5

344.2

126.8
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

224.8

False

32

1223.7
Chunk size (ms): 160
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

34.3

33.8

35.5

36.1

61.2

1.0

True

1

38.1

34.8

48.5

50.0

93.0

1.0

False

8

41.2

40.7

43.1

43.9

76.4

8.0

True

8

53.6

41.4

89.8

96.9

165.4

8.0

False

16

52.4

51.1

53.9

70.7

104.1

15.9

True

16

70.9

51.6

114.8

129.8

257.8

15.9

False

32

78.4

64.4

102.5

105.8

145.8

31.8

True

32

115.5

102.1

201.6

217.2

394.5

31.6

False

48

105.5

124.7

132.7

136.8

174.0

47.6

True

48

169.7

141.4

258.0

287.1

518.0

47.3
Chunk size (ms): 960
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

62.0

61.3

64.7

65.6

92.2

1.0

True

1

93.9

76.6

100.4

101.0

538.6

1.0

False

64

230.0

269.0

275.3

278.4

280.1

63.4

True

64

388.3

425.9

495.3

510.8

525.9

63.2

False

128

366.7

398.8

416.5

429.6

446.6

126.2

True

128

600.0

621.0

644.7

900.0

957.3

124.4
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

21.0

False

32

357.2

True

32

270.1
Chunk size (ms): 160
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

23.02

23.176

28.694

29.829

44.012

0.99863

8

34.593

33.966

40.212

45.607

94.882

7.9646

16

42.333

41.683

50.64

58.927

93.028

15.953

32

55.452

51.111

76.248

82.525

129.15

31.828

48

72.80

175.236

94.222

106.1

223.45

47.592

64

97.943

100.13

116.06

126.09

240.68

63.512
Chunk size (ms): 960
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

29.309

29.149

36.643

41.075

47.067

0.99879

64

114.55

116.94

159.24

177.71

189.42

63.655

128

170.28

173.34

217

220.87

306.83

126.76

256

265.46

262.12

374.34

445.03

610.07

251.12

384

322.1

300.5

478.52

627.84

962.12

374.99

512

437.49

385.16

733.84

1084.8

1529.9

493.42
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

166.71

False

32

505.29
Chunk size (ms): 160
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

17.854

15.285

20.322

21.213

28.771

0.99935

8

24.344

22.916

30.133

38.193

53.096

7.9913

16

33.628

31.514

39.454

61.332

77.594

15.975

32

51.488

51.29

60.141

99.328

125.21

31.915

48

66.051

66.906

78.255

106.77

150.16

47.831

64

70.315

75.312

85.973

123.24

183.86

63.743
Chunk size (ms): 960
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

21.612

20.434

25.3

26.234

62.83

26.234

64

95.964

87.699

181.02

183.42

188.01

189.01

128

174.07

152.57

280.41

346.19

356.8

189.01

256

281.56

249.97

523.14

594.08

682.59

700.86

384

392.68

336.15

758.24

870.87

1002.9

1033.1

512

540.34

437.2

1118.3

1210.2

1351.9

1424.3
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

337.94

False

32

3229.5
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

14.5

13.6

14.8

15.3

22.0

1.0

8

16.7

16.4

17.7

18.6

33.8

8.0

16

19.4

18.6

20.3

26.8

49.1

16.0

32

25.1

26.8

31.0

31.8

75.8

31.9

48

30.0

31.7

35.7

39.1

99.0

47.8

64

36.1

38.2

41.8

43.3

138.0

63.7
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

19.5

17.4

23.6

26.4

80.3

1.0

64

48.7

45.3

52.4

154.2

164.1

63.7

128

65.8

54.4

74.6

198.1

285.4

126.8

256

99.9

78.1

113.7

356.7

554.2

251.6

384

130.2

101.0

150.3

488.8

788.8

374.2

512

178.8

130.8

316.2

638.8

1106.0

494.6
Language model: n-gram

# of streams

Throughput (RTFX)

1

303.6

32

3331.5
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

38.3

18.6

29.5

87.6

536.4

1.0

8

25.4

22.3

33.9

34.4

81.2

7.9

16

37.7

35.7

56.4

58.3

123.2

15.8

32

38.1

37.7

62.2

65.1

136.0

31.5

48

46.1

46.1

70.2

73.0

142.9

47.1

64

58.5

57.0

84.7

87.0

275.5

62.4

128

75.2

64.1

114.9

120.0

415.4

122.1
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

66.0

40.9

49.0

338.8

338.8

1.0

64

111.4

117.9

124.8

273.5

275.2

62.4

128

152.1

144.6

166.0

363.3

479.8

122.5
Language model: n-gram

# of streams

Throughput (RTFX)

1

50.4

32

605.5

64

782.7

128

1004.9

# of streams

Throughput (RTFX)

Average Latency (ms)

1

208.9

272.85

32

2210.3

745.51

64

2601

810.1
Chunk size (ms): 320
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

1.0

16

15.7

32

31.0

64

60.9

128

117.3
Chunk size (ms): 1600
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

1.0

64

61.8

128

121.3
Language model: n-gram

# of streams

Throughput (RTFX)

1

28.9

32

615.3

32

782.2

64

803.9

128

1170.7
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 355

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

10

9.9

11.3

12

40

1

8

12.6

12

13.4

17

31

8

16

17

15

22

25

40

15.98

32

23

23

31

33

50

31.94

48

29

28

40

41

70

47.9

64

33.6

38

45

47

70

63.9

128

49

47

64

67

150

127.6

256

84

75

107

126

391

255
Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1400

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

14

11

20

40

80

1

64

39

40

55

80

110

63.9

128

58

50

75

150

202

127.6

256

90

80

115

240

380

255

384

120

107

155

316

530

381.4

512

149

130

196

400

700

508

768

258

200

630

680

1280

756

1024

420

263

1280

1350

1900

992
Language model: n-gram

# of streams

Throughput (RTFX)

32

467
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

90

False

32

370
Language model: none

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

110

False

32

1255
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

9.3408

9.3226

9.8487

9.9818

12.067

0.99961

False

8

12.606

10.263

10.845

13.946

30.926

7.9962

False

16

16.353

11.526

16.15

16.997

229.76

15.991

False

32

29.249

26.865

28.171

29.427

210.22

31.956

False

64

28.558

29.476

41.704

42.326

46.42

63.873
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

9.6244

9.5538

10.097

10.229

12.866

0.9996

False

8

14.665

10.621

11.351

15.111

118.39

7.9962

False

16

19.055

15.822

26.533

28.685

66.638

15.98

False

32

32.152

25.817

30.409

31.121

427.99

31.957

False

64

35.503

33.099

34.777

36.086

340.15

63.887

False

128

46.861

38.773

42.081

64.611

400.26

127.54

False

256

53.239

43.716

64.049

94.044

258.96

254.88

False

512

104.02

73.438

258.62

336.12

420.37

503.42
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

9.5612

9.4801

10.022

10.212

13.059

0.99959

False

8

15.512

10.506

11.199

15.543

136.69

7.9961

False

16

26.723

16.46

29.124

30.463

421.71

15.979

False

32

34.181

25.042

31.631

32.797

493.99

31.954

False

64

30.222

30.838

43.546

44.166

48.958

63.868
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

9.8389

9.7843

10.354

10.544

12.949

0.99958

False

8

15.668

10.909

15.178

16.21

146.66

7.9961

False

16

16.379

11.944

16.057

18.209

109.25

15.99

False

32

33.224

21.587

30.868

31.922

500.12

31.96

False

64

40.087

33.156

35.075

37.459

438.92

63.887

False

128

47.509

38.283

40.893

46.769

449.2

127.7

False

256

52.992

43.092

64.392

71.985

299.05

254.88

False

512

101.67

74.537

221.67

348.23

462

508.02
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 179

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

17.79

15.812

22.171

22.66

24.527

0.99925

8

19.619

18.702

20.283

21.283

49.858

7.9866

16

24.347

22.816

24.601

30.805

83.174

15.958

32

32.883

30.65

40.314

40.992

129.39

31.856

48

43.084

44.219

50.994

56.952

210.66

47.689

64

53.643

53.416

61.031

97.948

264.43

63.476
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 810

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

20.624

19.301

26.012

28.626

50.408

0.99918

64

73.596

71.06

84.034

234.8

251.47

63.497

128

123.56

110.62

139.56

300.04

449.06

126.28

256

188.12

162.21

200.33

538

814.35

249.61

384

268.43

198.76

527.84

786.32

1372.3

369.42

512

405.24

287.28

1347.4

1439.1

2252.5

486.61
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

318.26

False

32

2085

True

32

125
Chunk size (ms): 320
Language model: n-gram
Maximum effective # of streams with n-gram language model: 179

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

25.742

25.178

29.273

30.679

40.96

0.99891

8

37.458

36.717

43.875

45.592

57.38

7.9865

16

46.788

45.738

51.555

60.965

75.74

15.963

32

64.08

57.471

84.993

89.653

128.29

31.873

48

85.545

96.194

111.54

117.86

176.5

47.714

64

93.02

104.95

116

124.89

195.03

63.61
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 810

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

21.451

20.791

23.836

24.61

53.358

0.99922

64

91.55

103.46

124.84

126.52

134.28

63.575

128

177.23

190.8

213.12

218.6

244.28

127.01

256

279.71

279.51

358.52

371.5

449.47

252.36

384

386.16

389.76

521.21

556.27

722.57

375.28

512

492.63

496.77

691.83

793.61

1101.9

494.73
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

338.78

False

32

3041.9
Chunk size (ms): 160
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

41.4

41.0

43.1

43.9

65.2

1.0

8

69.5

68.9

73.8

76.8

116.3

8.0

16

84.7

80.4

108.1

113.5

149.9

15.9

32

138.2

147.3

172.9

180.1

232.4

31.7

48

2610.6

2456.5

4743.8

4941.9

6120.2

41.5
Chunk size (ms): 960
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

32.0

32.0

33.2

34.9

35.6

1.0

64

263.5

310.4

324.9

326.4

342.2

63.4

128

562.0

591.7

646.0

829.8

835.5

124.8
Language model: n-gram

# of streams

Throughput (RTFX)

1

151.5

32

599.4
Chunk size (ms): 160
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

51.9

51.2

53.3

54.1

91.2

1.0

True

1

57.0

51.5

74.3

75.6

134.2

1.0

False

8

78.6

77.6

82.8

84.0

144.3

8.0

True

8

92.7

80.3

127.8

131.8

246.6

7.9

False

16

85.0

83.8

86.4

87.3

165.2

15.9

True

16

107.9

85.5

161.3

164.5

350.9

15.8

False

32

147.0

149.3

176.1

184.5

295.1

31.7

True

32

273.1

241.1

415.1

505.6

817.4

31.2
Chunk size (ms): 960
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

74.0

73.7

75.3

77.5

94.4

1.0

True

1

108.0

96.6

100.2

111.7

473.1

1.0

False

64

366.4

427.7

438.1

447.2

456.1

63.1

True

64

541.1

604.0

658.0

803.4

833.1

62.4
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

18.7

False

32

252.4

True

32

148.7
Chunk size (ms): 160
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

30.948

31.181

34.543

36.056

47.975

0.99827

8

47.991

48.392

53.894

55.871

87.543

7.978

16

61.284

61.356

68.72

76.258

118.18

15.923

32

75.633

74.065

95.78

102.73

155.86

31.809

48

91.854

99.673

111.41

113.95

255.06

47.621

64

114.38

126.17

135.55

139.42

321.86

63.361
Chunk size (ms): 960
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

26.371

26.905

29.871

34.808

35.852

0.9987

64

116.95

132.38

156.75

165.34

170.12

63.681

128

227.25

232.36

279.57

295.28

372.04

126.56

256

351

363.49

448.89

506.45

769.04

249.55

384

451.4

451.33

622.64

676.79

935.07

372.85

512

579.83

578.14

838.03

1041.4

1447.4

489.7
Language model: n-gram

# of streams

Throughput (RTFX)

1

213.96

32

1021
Chunk size (ms): 160
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

25.898

25.159

29.92

31.529

38.217

0.99898

8

39.652

38.118

48.426

56.406

75.964

7.986

16

51.214

48.284

58.617

78.85

103.47

15.963

32

62.971

65.896

79.542

102.56

123.49

31.882

48

82.938

89.665

102.51

151.8

176.93

47.767

64

107.35

113.37

128.69

197.71

246.97

63.598
Chunk size (ms): 960
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

23.143

22.251

26.383

26.517

62.112

0.9993

64

123.91

120.69

209.79

221.48

222.74

63.746

128

250.18

224.35

404.34

470.45

495.53

127

256

424.4

391.15

726.09

848.57

940.01

250.66

384

644.41

550.55

1274

1344.7

1493.9

369.39

512

882.82

803.01

1746.6

1874.4

1955.3

486.89
Language model: n-gram

# of streams

Throughput (RTFX)

1

299.83

32

2790.2
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

26.7

26.5

27.8

28.3

46.2

1.0

8

32.1

31.1

32.5

33.3

75.2

8.0

16

39.2

35.2

53.3

54.4

102.4

15.9

32

47.4

38.6

61.9

64.3

140.2

31.8

48

58.2

64.3

68.8

74.1

188.8

47.7

64

73.2

77.7

81.6

119.6

282.5

63.5
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

30.4

30.0

31.3

37.2

37.8

1.0

64

100.0

99.9

117.6

256.4

276.0

63.5

128

137.8

109.5

164.0

355.9

468.1

126.2

256

218.3

175.4

281.1

611.2

892.1

249.0

384

350.6

280.9

838.8

937.2

1633.7

367.5

512

624.9

365.5

1666.0

2471.5

2800.5

481.6
Language model: n-gram

# of streams

Throughput (RTFX)

1

166.2

32

1555.0
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

70.0

38.8

51.9

285.2

693.8

1.0

8

42.7

37.7

53.4

54.2

134.4

7.9

16

50.4

44.2

64.2

67.1

193.3

15.7

32

61.7

56.4

95.7

98.9

207.7

31.2

48

71.0

69.2

109.0

115.3

278.3

46.5

64

88.1

93.1

129.3

136.4

341.7

61.3

128

162.9

149.1

243.1

252.6

769.8

117.8
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

83.9

57.0

67.5

382.3

382.3

1.0

64

165.8

180.7

206.0

386.2

387.1

61.8

128

240.4

229.1

293.1

566.6

722.3

120.0
Language model: n-gram

# of streams

Throughput (RTFX)

1

31.2

32

431.1

64

536.7

128

633.3

# of streams

Throughput (RTFX)

Average Latency (ms)

1

128.5

433.0

32

1326.0

1268.77
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 233

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

13

11.8

12.8

14

40

1

8

17.6

16.8

18.5

22

39

8

16

22.5

21.3

25

31

60.3

15.98

32

32.4

35

42

46

70

31.93

48

41

40

58

59

100

47.9

64

46

50

64

66

100

63.8

128

73

66

94

97

220

127.5
Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 980

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

16

13

20

40

80

1

64

60

60

80

110

180

63.8

128

90

80

110

230

300

127.5

256

133.3

120

174

340

530

254

384

183

166

245

430

800

380

512

260

223

510

600

1200

505

768

535

354

1500

1640

2150

739

1024

940

600

2300

2570

2930

960
Language model: n-gram

# of streams

Throughput (RTFX)

32

460
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

60

False

32

234
Language model: none

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

108.04

False

32

827.71
Chunk size (ms): 320
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

1.0

16

15.4

32

30.3

64

58.8

128

107.7
Chunk size (ms): 1600
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

1.0

64

60.7

128

117.3
Language model: n-gram

# of streams

Throughput (RTFX)

1

23.0

32

445.6

32

545.9

64

655.7

128

904.6
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

13.102

13.052

13.518

13.618

19.613

0.99938

False

8

17.258

14.713

15.891

16.816

31.669

7.994

False

16

22.773

17.01

18.957

29.031

210.83

15.985

False

32

32.441

31.072

34.966

36.057

355.53

31.946

False

64

36.871

36.746

51.367

52.308

83.872

63.826
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

14.44

14.352

15.026

15.157

21.936

0.99929

False

8

18.468

15.934

16.95

17.537

32.51

7.9933

False

16

22.79

18.733

32.64

33.727

48.82

15.976

False

32

34.267

21.227

37.192

38.529

334.03

31.944

False

64

44.112

43.585

45.904

49.415

345.29

63.849

False

128

67.936

58.998

62.494

111.11

432.4

127.53

False

256

74.346

61.015

87.088

134.62

251.33

254.47

False

512

527.34

561.98

774.91

833.24

937.56

500.9
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

13.994

13.831

14.547

14.808

21.6

0.99931

False

8

23.515

15.284

16.59

23.974

316.29

7.9938

False

16

22.393

17.427

19.237

31.029

154.72

15.977

False

32

41.227

31.802

36.331

45.725

579.61

31.944

False

64

36.7

36.802

51.946

52.998

69.072

63.822
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

15.29

15.18

15.871

15.974

23.805

0.99922

False

8

21.087

16.36

17.077

17.63

183.35

7.9928

False

16

34.187

18.991

20.461

33.655

543.73

15.985

False

32

85.397

35.526

38.658

457.3

1448.1

31.941

False

64

51.393

43.668

45.916

52.046

499.23

63.848

False

128

68.083

54.226

60.272

97.797

592.4

127.54

False

256

73.366

58.707

83.998

110.79

355.43

254.51
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 190

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

11.518

10.501

11.753

12.329

29.332

0.99948

8

13.042

12.727

14.303

16.54

27.45

7.9934

16

17.579

16.357

25.071

26.493

42.529

15.974

32

21.415

18.903

27.705

28.62

65.338

31.924

48

32.285

32.166

34.611

35.804

102.55

47.839

64

33.933

36.076

39.682

41.26

120.46

63.75
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 900

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

14.345

12.899

18.496

21.621

49.489

0.99941

64

43.724

41.908

48.17

138.95

140.61

63.715

128

76.158

69.027

80.239

198.79

277.37

126.88

256

113.72

89.307

128.73

294.93

488.96

251.9

384

150.8

133.5

170.69

465.93

722.34

374.76

512

198.83

173.53

280.75

577.5

975.18

495.82
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

365.61

False

32

3638

True

32

101.5
Chunk size (ms): 320
Language model: n-gram
Maximum effective # of streams with n-gram language model: 270

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

13.377

12.014

14.8

15.721

17.616

0.99944

8

20.362

19.784

23.525

26.147

33.041

7.9919

16

28.97

28.08

34.588

37.939

52.757

15.97

32

42.96

38.11

55.592

57.928

94.558

31.904

48

58.84

67.281

75.958

77.311

136.62

47.794

64

79.065

88.762

99.511

109.32

181.85

63.634
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1240

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

17.358

16.19

19.135

23.812

52.552

0.99925

64

86.236

102.67

110.26

112.32

119.81

63.754

128

204.03

205.92

220.27

223.14

250.96

126.93

256

315.08

321.68

395.18

408.04

502.56

251.93

384

423.9

421.51

577.25

664

857.63

373.82

512

573

563.93

874.58

1039.3

1263.3

492.08
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

335.32

False

32

2876.4
Chunk size (ms): 160
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

21.6

21.1

22.8

23.4

32.5

1.0

8

31.5

31.2

33.8

35.2

51.5

8.0

16

45.5

45.4

47.8

53.5

79.2

16.0

32

67.4

61.8

89.0

90.6

119.4

31.8

48

98.9

116.6

127.1

134.2

182.2

47.6
Chunk size (ms): 960
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

21.1

20.3

23.0

24.7

49.2

1.0

64

161.7

197.5

204.8

208.3

212.7

63.6

128

369.0

396.0

432.2

450.6

455.8

126.2
Language model: n-gram

# of streams

Throughput (RTFX)

1

264.5

32

882.6
Chunk size (ms): 160
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

29.2

28.8

30.4

31.0

51.3

1.0

True

1

42.0

29.2

42.7

44.9

432.5

1.0

False

8

38.9

38.4

40.9

41.5

72.6

8.0

True

8

45.0

39.6

58.2

60.3

119.8

8.0

False

16

55.1

53.9

56.9

69.1

114.0

15.9

True

16

68.7

54.3

93.4

117.3

235.3

15.9

False

32

85.0

68.0

111.5

114.5

158.0

31.8

True

32

115.1

108.5

188.4

205.2

376.7

31.6

False

48

127.5

148.5

158.0

191.3

265.8

47.6

True

48

213.6

204.7

283.7

343.9

555.8

47.3
Chunk size (ms): 960
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

59.8

59.2

61.8

63.2

92.3

1.0

True

1

85.1

72.2

76.2

83.5

514.8

1.0

False

64

255.4

304.4

310.3

313.2

315.0

63.4

True

64

372.3

422.1

463.4

469.2

471.6

63.1

False

128

478.7

513.0

528.6

666.1

686.7

125.3

True

128

687.7

695.9

776.2

1113.9

1620.6

123.7
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

20.7

False

32

336.4

True

32

260.5
Chunk size (ms): 160
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

19.671

20.104

21.685

21.96

38.539

0.99884

8

31.194

31.719

35.482

36.195

66.154

7.9835

16

45.007

46.339

50.019

51.456

92.321

15.953

32

61.018

56.473

77.184

79.764

136.4

31.801

48

79.726

87.697

98.868

100.64

172.36

47.647

64

102.58

117.05

125.58

130.69

271.66

63.453
Chunk size (ms): 960
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

24.776

24.027

30.257

32.001

63.361

0.99901

64

114.75

133.68

149.83

153.73

157.22

63.679

128

235.08

244.42

285.45

290.93

374.07

126.5

256

367.18

365.85

468.17

506.79

691.56

250.49

384

485.58

465.62

668.72

772.08

999.89

371.5

512

637.89

635.49

970.62

1132.4

1399.5

489.51
Language model: n-gram

# of streams

Throughput (RTFX)

1

180.37

32

1037.8
Chunk size (ms): 160
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

14.202

13.052

17.88

19.466

25.956

0.99947

8

24.437

22.858

29.938

38.885

49.618

7.9913

16

36.536

34.467

44.059

54.708

81.838

15.973

32

48.218

52.974

64.388

81.382

99.757

31.904

48

69.337

78.074

89.509

110.77

143.67

47.784

64

92.235

101.77

114.86

181.18

204.97

63.62
Chunk size (ms): 960
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

19.475

17.506

22.395

23.533

78.759

0.99942

64

120.23

118.25

207.22

215.26

240.07

63.754

128

246.07

227.94

373.71

448.51

456.11

126.96

256

426

422.2

724.17

801.8

895.13

251.23

384

572.76

545.31

976.51

1096.7

1214.7

372.22

512

781.89

704.9

1342

1422.6

1613.6

491.52
Language model: n-gram

# of streams

Throughput (RTFX)

1

349.89

32

2885.7
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

20.0

18.8

20.0

20.8

36.8

1.0

8

21.5

21.1

22.5

23.6

41.5

8.0

16

24.5

23.7

25.3

35.4

53.5

16.0

32

32.9

35.6

41.5

42.8

78.7

31.9

48

39.3

44.2

47.8

49.7

99.4

47.8

64

46.9

51.4

55.0

56.4

152.1

63.7
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

23.3

21.6

26.8

29.7

71.9

1.0

64

60.1

59.4

68.0

152.8

157.1

63.7

128

89.0

91.7

101.0

196.8

289.9

126.8

256

133.7

108.4

173.2

383.8

552.3

251.6

384

183.2

150.9

251.1

558.0

819.9

374.2

512

256.3

205.5

484.9

665.2

1351.4

494.5
Language model: n-gram

# of streams

Throughput (RTFX)

1

286.7

32

2490.4
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

21.8

21.2

22.1

22.4

39.6

1.0

True

1

25.6

22.5

35.0

35.5

71.3

1.0

False

8

27.5

26.9

28.2

29.7

61.9

8.0

True

8

34.7

28.2

48.4

50.4

120.4

8.0

False

16

36.9

35.1

36.6

57.6

97.4

15.9

True

16

55.2

56.0

82.7

84.6

193.7

15.9

False

32

51.6

39.9

65.7

68.2

131.3

31.8

True

32

71.6

64.6

146.0

150.3

303.5

31.7

False

48

68.0

76.7

85.8

92.2

168.5

47.7

True

48

101.9

83.0

178.4

189.3

479.8

47.3
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

55.8

55.5

56.3

61.4

61.8

1.0

True

1

70.0

69.2

70.6

79.2

80.0

1.0

False

64

175.2

194.2

197.8

273.0

312.9

63.5

True

64

263.9

292.5

313.6

458.7

465.9

63.1

False

128

252.2

262.2

273.2

397.9

471.7

126.1

False

256

438.2

419.6

492.1

819.3

1027.5

248.0

False

384

759.5

626.9

1600.5

1968.1

2799.9

364.6

False

512

2054.4

1823.7

3943.4

4720.0

5667.5

456.1

False

512

2015.4

1795.2

3924.8

4561.5

5507.2

457.1
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

21.9

True

1

19.9

False

32

420.4

True

32

308.5

Speaker Diarization

# of streams

Throughput (RTFX)

Average Latency (ms)

False

1

158.3

352.24

False

32

1631.3

1018.84
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 280

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

11

10.3

11.2

12.4

30

1

8

20

19

26

30

42

7.99

16

28

26

35

40

56

15.97

32

35

35

48

52

73

31.9

64

50

55

66

70

100

63.8
Chunk size (ms): 800
Language model: n-gram
Maximum effective # of streams with n-gram language model: 1180

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

14

11.5

20

30

60

1

64

70

70

90

100

170

63.8

128

88

84

110

190

250

127.4

256

128

117

164

300

460

254.4
Language model: n-gram

# of streams

Throughput (RTFX)

32

440
Language model: n-gram

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

70

False

32

193.5
Language model: none

Speaker Diarization

# of streams

Throughput (RTFX)

False

1

6.2

False

32

43.3
Chunk size (ms): 320
Language model: n-gram

# of streams

Throughput (RTFX)

Average Latency (ms)

1

1.0

86.53

8

7.9

136.34

16

15.8

163.55

32

31.4

253.70

48

44.8

991.17

64

58.9

1180.73
Chunk size (ms): 1600
Language model: n-gram

# of streams

Throughput (RTFX)

Average Latency (ms)

1

1.0

87.29

64

63.0

433.03

128

125.0

586.62

256

246.3

836.96

384

337.1

2274.47

512

342.7

7912.27
Language model: n-gram

# of streams

Throughput (RTFX)

Average Latency (ms)

1

85.2

642.87

32

1056.6

1606.57
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

9.2825

9.2558

9.7189

9.877

12.139

0.99961

False

8

12.442

10.341

10.88

14.296

21.05

7.9962

False

16

13.595

11.598

12.382

17.042

23.913

15.991

False

32

20.704

13.185

21.599

22.169

192.45

31.968

False

64

21.558

22.259

29.916

30.413

40.959

63.893
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

9.5904

9.5805

10.03

10.13

12.626

0.99959

False

8

12.604

10.6

11.21

12.336

19.945

7.9961

False

16

14.049

12.08

12.743

13.884

21.134

15.99

False

32

21.38

21.136

22.538

22.938

180.87

31.967

False

64

27.424

26.839

28.299

29.345

218.24

63.905

False

128

39.738

36.041

38.867

44.854

256.34

127.69

False

256

49.669

41.58

64.197

66.488

171.87

254.84

False

512

95.147

84.996

196.18

235.61

302.72

507.8
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

9.956

9.8412

10.352

10.484

14.014

0.99957

False

8

15.234

10.859

11.434

11.999

150.2

7.9939

False

16

16.302

11.95

12.689

19.337

136.82

15.991

False

32

28.829

13.908

22.441

24.082

526.41

31.968

False

64

22.279

22.46

30.846

31.389

41.178

63.892
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

9.8085

9.7295

10.271

10.359

13.166

0.99958

False

8

14.27

10.778

11.337

11.643

95.817

7.996

False

16

15.924

11.872

13.11

19.325

107.6

15.99

False

32

27.327

20.74

22.379

23.974

470.71

31.968

False

64

33.188

25.974

27.646

30.741

435.07

63.906

False

128

43.511

34.187

36.993

43.016

436.2

127.7

False

256

51.718

39.483

60.158

80.29

322.12

254.87

False

512

97.185

71.856

213.43

324.14

443.13

508.04
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

17.193

17.118

18.474

18.737

24.501

0.99914

False

8

20.655

18.849

21.041

29.132

35.76

7.9922

False

16

25.218

22.468

33.927

35.791

55.498

15.976

False

32

37.635

39.186

43.99

44.884

162.19

31.931

False

64

46.373

45.276

63.929

65.918

89.818

63.756
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

17.247

17.096

18.381

18.787

25.846

0.99919

False

8

20.673

18.85

20.5

21.691

35.584

7.9925

False

16

24.218

21.409

32.541

35.604

56.44

15.971

False

32

34.918

27.125

43.446

45.56

193.36

31.939

False

64

51.064

53.576

60.069

63.47

228.19

63.793

False

128

90.28

85.041

93.896

156.95

334.07

126.92

False

256

224.58

184.82

394.92

443.95

507.15

252.21
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

16.457

16.26

17.851

18.471

21.308

0.99916

False

8

21.093

18.523

20.572

21.338

35.941

7.9915

False

16

29.195

21.938

34.861

36.646

314.34

15.972

False

32

39.632

38.226

43.118

45.143

332.82

31.94

False

64

43.661

44.879

62.258

64.532

83.038

63.782
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

18.296

18.236

19.733

20.147

26.085

0.99911

False

8

24.647

19.265

34.339

35.553

48.723

7.9883

False

16

29.183

22.906

35.118

38.678

297.8

15.981

False

32

41.677

41.333

46.059

48.419

320.74

31.925

False

64

54.926

54.679

60.648

64.036

351.57

63.797

False

128

88.528

81.505

90.739

110.12

470.27

127.29
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

21.4

21.0

22.2

22.6

35.0

1.0

8

28.1

27.0

28.5

33.3

63.3

8.0

16

39.8

38.8

50.0

52.0

100.0

15.9

32

60.2

68.4

74.7

76.4

182.3

31.8

48

87.7

97.8

103.9

125.2

278.2

47.6

64

133.6

130.1

150.9

311.0

616.6

63.3
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

22.8

21.7

26.7

30.4

48.7

1.0

64

143.5

155.1

166.9

303.0

355.6

63.3

128

246.4

206.7

309.9

479.8

606.6

125.6

256

514.9

421.4

1014.2

1355.0

1910.0

245.0

384

1672.4

960.7

4440.0

5127.3

5731.1

355.7

512

6575.1

6269.7

12263.0

13473.0

17069.0

367.8
Language model: n-gram

# of streams

Throughput (RTFX)

1

224.3

32

994.4
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

50.8

28.9

39.4

160.6

583.1

1.0

8

39.0

35.1

48.2

51.2

135.1

7.9

16

55.6

48.7

68.8

72.2

239.2

15.7

32

79.6

71.0

95.7

127.1

347.0

30.9

48

114.8

116.5

170.8

188.0

482.4

45.5

64

177.8

172.6

234.3

247.5

709.6

59.5

128

4827.4

4660.0

8580.3

9186.2

10417.0

64.8
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

71.6

50.9

54.1

313.6

313.6

1.0

64

236.3

244.1

309.8

521.3

522.9

61.0

128

446.8

425.9

604.8

1075.0

1332.1

114.0
Language model: n-gram

# of streams

Throughput (RTFX)

1

34.0

32

249.0

64

240.6

128

218.4
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

11.055

11.069

11.592

11.72

13.727

0.99953

False

8

15.208

12.447

18.884

19.312

23.021

7.9936

False

16

18.335

17.055

20.511

21.309

51.751

15.985

False

32

20.255

18.446

21.833

22.727

93.102

31.959

False

64

20.132

19.973

25.508

26.904

36.769

63.863
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

11.023

10.966

11.484

11.703

13.471

0.99951

False

8

18.122

13.391

19.058

19.635

120.22

7.9936

False

16

19.924

17.675

20.364

21.006

99.018

15.986

False

32

21.109

19.086

22.617

23.741

106.68

31.958

False

64

25.153

22.195

26.912

28.189

128.97

63.896

False

128

29.852

26.194

32.747

35.396

130.52

127.69

False

256

35.663

27.792

34.833

65.13

273.43

253.19

False

512

42.337

27.338

94.553

166.36

227.06

508
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

11.13

11.088

11.676

11.821

14.332

0.99951

False

8

17.341

13.334

19.237

19.909

30.216

7.9931

False

16

22.795

18.31

20.933

21.792

266.67

15.984

False

32

23.167

18.738

22.424

23.808

258.07

31.959

False

64

21.205

20.616

27.661

30.442

39.193

63.876
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

11.184

11.079

11.745

11.955

14.54

0.99951

False

8

17.582

13.621

19.779

20.277

25.856

7.9932

False

16

20.834

18.651

20.703

21.368

186.09

15.984

False

32

22.061

18.788

22.72

23.809

195.32

31.956

False

64

24.566

20.759

25.165

26.359

237.58

63.879

False

128

30.744

23.492

28.869

31.171

379.07

127.66

False

256

31.362

25.418

32.511

47.525

223.83

254.83

False

512

38.724

26.331

60.814

133.12

267.8

508.13
Chunk size (ms): 320
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

1.0

16

15.6

32

31.1

64

61.5

128

119.1
Chunk size (ms): 1600
Language model: n-gram

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

1.0

64

62.3

128

122.8
Language model: n-gram

# of streams

Throughput (RTFX)

1

27.3

32

743.5

32

876.3

64

1008.0

128

1463.5
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

15.6

14.9

16.5

17.2

21.1

1.0

8

19.0

16.9

24.2

25.0

39.4

8.0

16

25.8

25.2

31.9

34.8

81.3

15.9

32

28.5

26.6

36.9

42.6

94.3

31.9

48

29.9

27.5

39.0

47.3

109.5

47.8

64

33.6

30.6

43.4

56.0

138.4

63.7
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

18.6

16.8

23.9

26.7

58.4

1.0

64

54.4

34.7

96.2

192.8

231.3

63.5

128

63.5

35.5

127.7

226.8

411.3

126.2

256

104.6

46.7

243.5

442.4

820.6

249.2

384

156.9

46.2

482.4

727.2

1339.6

369.5

512

227.1

48.5

847.0

1340.9

1724.3

486.0
Language model: n-gram

# of streams

Throughput (RTFX)

1

359.9

32

1986.6
Chunk size (ms): 160
Language model: n-gram
Maximum effective # of streams with n-gram language model: 160

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

39.3

19.9

25.9

106.7

554.0

1.0

8

23.6

21.5

28.6

29.7

68.6

7.9

16

31.5

31.9

46.9

48.1

76.1

15.8

32

39.2

37.7

54.5

63.4

147.7

31.5

48

47.6

43.6

65.6

71.0

205.2

47.1

64

52.5

47.1

72.9

82.0

227.8

62.6

128

67.6

57.9

94.0

117.4

334.2

122.3
Chunk size (ms): 960
Language model: n-gram
Maximum effective # of streams with n-gram language model: 770

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

1

68.1

32.7

41.0

447.6

447.6

1.0

64

144.5

137.8

168.4

336.1

345.7

62.0

128

161.2

138.6

231.1

382.9

596.0

121.1
Language model: n-gram

# of streams

Throughput (RTFX)

1

53.4

32

765.8

64

970.1

128

1198.5
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

13.288

13.175

13.768

13.857

17.618

0.99942

False

8

22.765

22.148

27.469

28.253

36.962

7.9897

False

16

27.616

25.759

28.601

29.731

197.58

15.978

False

32

30.299

25.644

30.368

31.426

279.64

31.941

False

64

29.339

29.314

36.982

43.673

59.146

63.812
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

11.771

11.727

12.302

12.54

14.898

0.9995

False

8

18.679

14.216

20.838

21.336

62.044

7.9925

False

16

21.07

19.147

22.5

24.453

66.873

15.981

False

32

22.814

20.3

24.51

25.264

141.84

31.951

False

64

30.741

26.497

33.98

35.489

227.84

63.836

False

128

38.402

27.386

36.828

41.375

415.54

127.51

False

256

36.199

30.04

41.576

72.86

219.8

254.37

False

512

63.551

37.646

156.08

228.9

334.06

506.51
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

13.353

13.294

13.767

13.894

18.243

0.99936

False

8

24.417

22.83

27.518

28.152

97.028

7.9896

False

16

32.187

25.366

29.73

31.086

404.76

15.976

False

32

33.082

24.612

31.154

32.677

416.11

31.939

False

64

30.827

29.356

44.918

46.294

61.043

63.816
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

10.578

10.063

10.897

13.468

27.683

0.99956

False

8

14.938

10.816

14.168

19.1

81.862

7.9957

False

16

21.912

12.326

23.051

24.997

438.32

15.99

False

32

28.826

23.392

28.124

37.838

396.86

31.963

False

64

35.45

29.05

33.536

45.803

409.96

63.896

False

128

43.818

35.492

46.663

56.873

421.98

127.7

False

256

53.328

39.75

61.402

100.6

370.01

254.77

False

512

89.143

69.274

180.11

282.69

404.79

507.99
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

9.8396

9.7895

10.384

10.634

11.574

0.99961

False

8

15.846

10.905

15.964

16.59

168.66

7.9944

False

16

14.694

12.398

17.721

18.742

22.671

15.988

False

32

20.992

20.263

21.692

22.273

176.18

31.969

False

64

21.402

21.466

27.939

28.668

36.173

63.904
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

10.094

10.047

10.753

10.996

11.797

0.9996

False

8

14.461

10.807

11.723

12.481

174.48

7.9965

False

16

15.315

12.39

17.978

18.406

23.724

15.988

False

32

21.711

20.141

21.942

22.724

184.32

31.968

False

64

26.646

25.379

26.969

27.953

155.29

63.908

False

128

33.997

29.878

34.462

38.475

241.22

127.73

False

256

46.309

37.191

56.038

67.967

256.08

254.97

False

512

77.809

68.526

146.54

209.63

293.6

508.48
Chunk size (ms): 160
Batch size: 32
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

9.9756

9.8858

10.726

10.871

11.934

0.9996

False

8

14.571

10.801

11.972

16.886

96.086

7.9963

False

16

16.703

12.381

18.524

19.127

93.091

15.991

False

32

25.623

20.263

22.047

23.229

388.95

31.969

False

64

22.756

22.167

28.912

29.44

37.225

63.898
Chunk size (ms): 160
Batch size: 128
Language model: n-gram

Speaker Diarization

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

False

1

10.204

10.125

10.793

10.944

12.188

0.9996

False

8

15.246

11.253

12.366

17.017

115.65

7.9961

False

16

23.576

12.832

19.121

19.747

461.9

15.986

False

32

27.63

20.782

22.619

23.967

446.14

31.969

False

64

32.681

25.973

27.889

30.569

419.83

63.908

False

128

40.901

31.628

35.04

39.622

416.51

127.71

False

256

46.568

36.53

56.05

65.101

290.98

254.98

False

512

90.362

67.627

188.48

327.7

463

508.52

On-Prem Hardware Specifications#

GPU

NVIDIA DGX A100 40GB

CPU

Model

AMD EPYC 7742 64-Core Processor

Thread(s) per core

2

Socket(s)

2

Core(s) per socket

64

NUMA node(s)

8

Frequency boost

enabled

CPU max MHz

2250

CPU min MHz

1500

RAM

Model

Micron DDR4 36ASF8G72PZ-3G2B2 3200MHz

Configured Memory Speed

2933 MT/s

RAM Size

32x64GB (2048GB Total)

GPU

NVIDIA H100 80GB HBM3

CPU

Model

Intel(R) Xeon(R) Platinum 8480CL

Thread(s) per core

2

Socket(s)

2

Core(s) per socket

56

NUMA node(s)

2

CPU max MHz

3800

CPU min MHz

800

RAM

Model

Micron DDR5 MTC40F2046S1RC48BA1 4800MHz

Configured Memory Speed

4400 MT/s

RAM Size

32x64GB (2048GB Total)

GPU

NVIDIA L40S

CPU

Model

AMD EPYC 7763 64-Core Processor

Thread(s) per core

1

Socket(s)

2

Core(s) per socket

64

NUMA node(s)

8

Frequency boost

enabled

CPU max MHz

3529

CPU min MHz

1500

RAM

Model

Samsung DDR4 M393A4K40DB3-CWE 3200MHz

Configured Memory Speed

3200 MT/s

RAM Size

16x32GB (512GB Total)

Model Accuracy#

ASR models are evaluated with the following metrics. Lower values indicate better accuracy, with 0% representing perfect transcription.

  • Word Error Rate (WER): Used for word-based languages (English, Spanish, and French). Measures the minimum word substitutions, insertions, and deletions needed to match the reference transcript, divided by total reference words.

  • Character Error Rate (CER): Used for character-based languages (Chinese, Japanese, and Mandarin). Measures the minimum character edits needed, divided by total reference characters.

  • Concatenated minimum-Permutation Word Error Rate (cpWER): Used for speaker diarization. Calculated by:

    1. Concatenating all utterances per speaker for both reference and hypothesis.

    2. Computing WER across all possible speaker permutations of the hypothesis.

    3. Selecting the lowest WER (best permutation).

Model Name

Language

Dataset

Best latency WER (%) ⬇️

Best throughput WER (%) ⬇️

Offline WER (%) ⬇️

Parakeet 1.1b CTC

en-US

MCV 7.1 test set

10.45

8.80

7.96

en-US

LibriSpeech test-other

6.34

4.74

4.09

en-US

CallHome (CH109)

46.09

41.35

39.61

en-US (Silero VAD)

LibriSpeech test-other

5.57

4.8

4.5

en-US (Telephony)

LibriSpeech test-other

7.33

5.11

4.17

en-US (Telephony)

CallHome (CH109)

30.13

27.82

28.91

en-US (Telephony) + Sortformer Diarizer

CallHome (CH109)

28.43 (cpWER)

-

-

Parakeet 0.6b TDT

en-US

AMI

-

-

11.46

en-US

Earnings22

-

-

11.65

en-US

Gigaspeech

-

-

9.15

en-US

LibriSpeech test-clean

-

-

2.01

en-US

LibriSpeech test-other

-

-

3.51

en-US

SPGISpeech

-

-

2.16

en-US

Tedlium

-

-

3.38

en-US

Voxpopuli

-

-

6.6

Parakeet 1.1b RNNT

en-US

MCV 7.1 test set

10.74

10.54

9.77

es-US

MLS test set

7.19

5.26

3.83

es-ES

Mediaspeech

16.15

14.42

11.51

fr-FR

MLS test set

11.41

9.10

6.36

de-DE

MLS test set

11.29

9.16

7.09

ru-RU

RuLS test set

21.44

19.23

17.39

Parakeet 0.6b CTC

en-US

MCV 7.1 test set

10.57

8.87

8.45

Parakeet 0.6b CTC

vi-VN

FLEURS Vietnamese test set

10

8.58

7.97

Parakeet 0.6b CTC

zh-CN

AISHELL1 & 2

5.81

5.84

6.09

Parakeet 0.6b CTC

es-US

MLS test set

9.14

6.15

5.34

Canary 1b

en-US

MCV 7.1 test set

Not supported

Not supported

6.78

es-US

MLS test set

Not supported

Not supported

3.54

de-DE

MLS test set

Not supported

Not supported

5.18

fr-FR

MLS test set

Not supported

Not supported

4.21

ru-RU

MCV 7.0 test set

Not supported

Not supported

10.33

es-ES

Mediaspeech

Not supported

Not supported

14.40

pt-BR

MCV 10.0 test set

Not supported

Not supported

5.83

Conformer 120m CTC

es-US

MCV 7.1 test set

6.75

6.26

5.66