Performance#

Evaluation Process#

This section shows the latency and throughput numbers for streaming and offline configurations of the Riva ASR service on different GPUs. These numbers were captured after the preconfigured ASR pipelines from our Quick Start scripts were deployed. The Jasper, QuartzNet, Conformer, and Citrinet-1024 acoustic models were tested.

In streaming mode, the client and the server used audio chunks of the same duration (100ms, 160ms, 800ms depending on the server configuration). The Riva streaming client riva_streaming_asr_client, provided in the Riva client image, was used with the --simulate_realtime flag to simulate transcription from a microphone, where each stream was doing three iterations over a sample audio file from the Librispeech dataset (1272-135031-0000.wav). The command used to measure performance was:

riva_streaming_asr_client \
   --chunk_duration_ms=<chunk_duration> \
   --simulate_realtime=true \
   --automatic_punctuation=true \
   --num_parallel_requests=<num_streams> \
   --word_time_offsets=true \
   --print_transcripts=false \
   --interim_results=false \
   --num_iterations=<3*num_streams> \
   --audio_file=1272-135031-0000.wav \
   --output_filename=/tmp/output.json

The riva_streaming_asr_client returns the following latency measurements:

  • intermediate latency: latency of responses returned with is_final == false

  • final latency: latency of responses returned with is_final == true

  • latency: the overall latency of all returned responses. This is what is tabulated in the tables below.

In offline mode, the command used to measure maximum throughput was:

riva_asr_client \
   --automatic_punctuation=true \
   --num_parallel_requests=32 \
   --word_time_offsets=true \
   --print_transcripts=false \
   --num_iterations=96 \
   --audio_file=5x_1272-135031-0000.wav \
   --output_filename=/tmp/output.json

Results#

Latencies and throughput measurements for streaming and offline configurations are reported in the following tables. Throughput is measured in RTFX (duration of audio generated / computation time).

Note

If the language model is none, the inference is performed with a greedy decoder. If the language model is n-gram, then a beam decoder was used.

For specifications of the hardware on which these measurements were collected, refer to the Hardware Specifications section.

Chunk size (ms): 160
Maximum effective # of streams with n-gram language model: 251
Maximum effective # of streams without language model: 270

Language model

# of streams

Latency (ms)

Throughput (RTFX)

avg

p50

p90

p95

p99

n-gram

1

10.49

10.207

10.60

11.46

16.5

0.999433

n-gram

8

14.68

14.16

15.23

17.2

33.1

7.9910

n-gram

16

27.2

27.3

29.7

30.6

49.4

15.9710

n-gram

32

39.5

38.6

41.1

42.7

78.67

31.9133

n-gram

48

48.5

47.6

50.8

52.3

102

47.833

n-gram

64

57.8

56.5

62.3

64.8

118

63.69

none

1

9.95

9.60

9.87

12.3

14.86

0.999543

none

8

13.7

13.1

14.1

16.6

25.3

7.9928

none

16

24.17

26.4

27.6

28.1

44.1

15.9747

none

32

37.0

36.5

37.9

39.6

66.0

31.924

none

48

44.4

43.7

47.7

49.9

87

47.854

none

64

50.3

49.5

55.8

59

105.7

63.766

Hardware Specifications#

GPU

NVIDIA DGX A100 40 GB

CPU

Model

AMD EPYC 7742 64-Core Processor

Thread(s) per core

2

Socket(s)

2

Core(s) per socket

64

NUMA node(s)

8

Frequency boost

enabled

CPU max MHz

2250

CPU min MHz

1500

RAM

Model

Micron DDR4 36ASF8G72PZ-3G2B2 3200MHz

Configured Memory Speed

2933 MT/s

RAM Size

32x64GB (2048GB Total)