riva/proto/riva_asr.proto#

service RivaSpeechRecognition

The RivaSpeechRecognition service provides two mechanisms for converting speech to text.

rpc RecognizeResponse Recognize(RecognizeRequest)

Recognize expects a RecognizeRequest and returns a RecognizeResponse. This request will block until the audio is uploaded, processed, and a transcript is returned.

rpc stream StreamingRecognizeResponse StreamingRecognize(stream StreamingRecognizeRequest)

StreamingRecognize is a non-blocking API call that allows audio data to be fed to the server in chunks as it becomes available. Depending on the configuration in the StreamingRecognizeRequest, intermediate results can be sent back to the client. Recognition ends when the stream is closed by the client.

rpc RivaSpeechRecognitionConfigResponse GetRivaSpeechRecognitionConfig(RivaSpeechRecognitionConfigRequest)

Enables clients to request the configuration of the current ASR service, or a specific model within the service.

message EndpointingConfig

EndpointingConfig is used for configuring different fields related to start or end of utterance

int32 start_history

start_history is the size of the window, in milliseconds, used to detect start of utterance. start_threshold is the percentage threshold used to detect start of utterance. (0.0 to 1.0) If start_threshold of start_history ms of the acoustic model output have non-blank tokens, start of utterance is detected.

optional

float start_threshold

optional

int32 stop_history

stop_history is the size of the window, in milliseconds, used to detect end of utterance. stop_threshold is the percentage threshold used to detect end of utterance. (0.0 to 1.0) If stop_threshold of stop_history ms of the acoustic model output have non-blank tokens, end of utterance is detected and decoder will be reset.

optional

float stop_threshold

optional

int32 stop_history_eou

stop_history_eou and stop_threshold_eou are used for 2-pass end of utterance. stop_history_eou is the size of the window, in milliseconds, used to trigger 1st pass of end of utterance and generate a partial transcript with stability of 1. (stop_history_eou < stop_history) stop_threshold_eou is the percentage threshold used to trigger 1st pass of end of utterance. (0.0 to 1.0) If stop_threshold_eou of stop_history_eou ms of the acoustic model output have non-blank tokens, 1st pass of end of utterance is triggered.

optional

float stop_threshold_eou

optional

message RecognitionConfig

Provides information to the recognizer that specifies how to process the request

nvidia.riva.AudioEncoding encoding

The encoding of the audio data sent in the request.

All encodings support only 1 channel (mono) audio.

int32 sample_rate_hertz

The sample rate in hertz (Hz) of the audio data sent in the RecognizeRequest or StreamingRecognizeRequest messages. The Riva server will automatically down-sample/up-sample the audio to match the ASR acoustic model sample rate. The sample rate value below 8kHz will not produce any meaningful output.

string language_code

Required. The language of the supplied audio as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt) language tag. Example: “en-US”.

int32 max_alternatives

Maximum number of recognition hypotheses to be returned. Specifically, the maximum number of SpeechRecognizeAlternative messages within each SpeechRecognizeResult. The server may return fewer than max_alternatives. If omitted, will return a maximum of one.

bool profanity_filter

A custom field that enables profanity filtering for the generated transcripts. If set to ‘true’, the server filters out profanities, replacing all but the initial character in each filtered word with asterisks. For example, “x**”. If set to false or omitted, profanities will not be filtered out. The default is false.

SpeechContext speech_contexts(repeated)

Array of SpeechContext. A means to provide context to assist the speech recognition. For more information, see SpeechContext section

int32 audio_channel_count

The number of channels in the input audio data. If 0 or omitted, defaults to one channel (mono). Note: Only single channel audio input is supported as of now.

bool enable_word_time_offsets

If true, the top result includes a list of words and the start and end time offsets (timestamps), and confidence scores for those words. If false, no word-level time offset information is returned. The default is false.

bool enable_automatic_punctuation

If ‘true’, adds punctuation to recognition result hypotheses. The default ‘false’ value does not add punctuation to result hypotheses.

bool enable_separate_recognition_per_channel

This needs to be set to true explicitly and audio_channel_count > 1 to get each channel recognized separately. The recognition result will contain a channel_tag field to state which channel that result belongs to. If this is not true, we will only recognize the first channel. The request is billed cumulatively for all channels recognized: audio_channel_count multiplied by the length of the audio. Note: This field is not yet supported.

string model

Which model to select for the given request. If empty, Riva will select the right model based on the other RecognitionConfig parameters. The model should correspond to the name passed to riva-build with the –name argument

bool verbatim_transcripts

The verbatim_transcripts flag enables or disable inverse text normalization. ‘true’ returns exactly what was said, with no denormalization. ‘false’ applies inverse text normalization, also this is the default

SpeakerDiarizationConfig diarization_config

Config to enable speaker diarization and set additional parameters. For non-streaming requests, the diarization results will be provided only in the top alternative of the FINAL SpeechRecognitionResult.

RecognitionConfig.CustomConfigurationEntry custom_configuration (repeated)

Custom fields for passing request-level configuration options to plugins used in the model pipeline.

EndpointingConfig endpointing_config

Config for tuning start or end of utterance parameters. If empty, Riva will use default values or custom values if specified in riva-build arguments.

optional

message RecognitionConfig.CustomConfigurationEntry
string key
string value
message RecognizeRequest

RecognizeRequest is used for batch processing of a single audio recording.

RecognitionConfig config

Provides information to recognizer that specifies how to process the request.

bytes audio

The raw audio data to be processed. The audio bytes must be encoded as specified in RecognitionConfig.

nvidia.riva.RequestId id

The ID to be associated with the request. If provided, this will be returned in the corresponding response.

message RecognizeResponse

The only message returned to the client by the Recognize method. It contains the result as zero or more sequential SpeechRecognitionResult messages.

SpeechRecognitionResult results(repeated)

Sequential list of transcription results corresponding to sequential portions of audio. Currently only returns one transcript.

nvidia.riva.RequestId id

The ID associated with the request

message RivaSpeechRecognitionConfigRequest
string model_name

If model is specified only return config for model, otherwise return all configs.

message RivaSpeechRecognitionConfigResponse
RivaSpeechRecognitionConfigResponse.Config model_config (repeated)
message RivaSpeechRecognitionConfigResponse.Config
string model_name
RivaSpeechRecognitionConfigResponse.Config.ParametersEntry parameters (repeated)
message RivaSpeechRecognitionConfigResponse.Config.ParametersEntry
string key
string value
message SpeakerDiarizationConfig

Config to enable speaker diarization.

bool enable_speaker_diarization

If ‘true’, enables speaker detection for each recognized word in the top alternative of the recognition result using a speaker_tag provided in the WordInfo.

int32 max_speaker_count

Maximum number of speakers in the conversation. This gives flexibility by allowing the system to automatically determine the correct number of speakers. If not set, the default value is 8.

message SpeechContext

Provides “hints” to the speech recognizer to favor specific words and phrases in the results.

string phrases(repeated)

A list of strings containing words and phrases “hints” so that the speech recognition is more likely to recognize them. This can be used to improve the accuracy for specific words and phrases, for example, if specific commands are typically spoken by the user. This can also be used to add additional words to the vocabulary of the recognizer.

float boost

Hint Boost. Positive value will increase the probability that a specific phrase will be recognized over other similar sounding phrases. The higher the boost, the higher the chance of false positive recognition as well. Though boost can accept a wide range of positive values, most use cases are best served with values between 0 and 20. We recommend using a binary search approach to finding the optimal value for your use case.

message SpeechRecognitionAlternative

Alternative hypotheses (a.k.a. n-best list).

string transcript

Transcript text representing the words that the user spoke.

float confidence

The confidence estimate. A higher number indicates an estimated greater likelihood that the recognized word is correct. This field is set only for a non-streaming result or, for a streaming result where is_final=true. This field is not guaranteed to be accurate and users should not rely on it to be always provided. Although confidence can currently be roughly interpreted as a natural-log probability, the estimate computation varies with difference configurations, and is subject to change. The default of 0.0 is a sentinel value indicating confidence was not set.

WordInfo words(repeated)

A list of word-specific information for each recognized word. Only populated if is_final=true

message SpeechRecognitionResult

A speech recognition result corresponding to the latest transcript

SpeechRecognitionAlternative alternatives(repeated)

May contain one or more recognition hypotheses (up to the maximum specified in max_alternatives). These alternatives are ordered in terms of accuracy, with the top (first) alternative being the most probable, as ranked by the recognizer.

int32 channel_tag

For multi-channel audio, this is the channel number corresponding to the recognized result for the audio from that channel. For audio_channel_count = N, its output values can range from ‘1’ to ‘N’.

float audio_processed

Length of audio processed so far in seconds

message StreamingRecognitionConfig

Provides information to the recognizer that specifies how to process the request

RecognitionConfig config

Provides information to the recognizer that specifies how to process the request

bool interim_results

If true, interim results (tentative hypotheses) may be returned as they become available (these interim results are indicated with the is_final=false flag). If false or omitted, only is_final=true result(s) are returned.

message StreamingRecognitionResult

A streaming speech recognition result corresponding to a portion of the audio that is currently being processed.

SpeechRecognitionAlternative alternatives(repeated)

May contain one or more recognition hypotheses (up to the maximum specified in max_alternatives). These alternatives are ordered in terms of accuracy, with the top (first) alternative being the most probable, as ranked by the recognizer.

bool is_final

If false, this StreamingRecognitionResult represents an interim result that may change. If true, this is the final time the speech service will return this particular StreamingRecognitionResult, the recognizer will not return any further hypotheses for this portion of the transcript and corresponding audio.

float stability

An estimate of the likelihood that the recognizer will not change its guess about this interim result. Values range from 0.0 (completely unstable) to 1.0 (completely stable). This field is only provided for interim results (is_final=false). The default of 0.0 is a sentinel value indicating stability was not set.

int32 channel_tag

For multi-channel audio, this is the channel number corresponding to the recognized result for the audio from that channel. For audio_channel_count = N, its output values can range from ‘1’ to ‘N’.

float audio_processed

Length of audio processed so far in seconds

message StreamingRecognizeRequest

A StreamingRecognizeRequest is used to configure and stream audio content to the Riva ASR Service. The first message sent must include only a StreamingRecognitionConfig. Subsequent messages sent in the stream must contain only raw bytes of the audio to be recognized.

StreamingRecognitionConfig streaming_config

Provides information to the recognizer that specifies how to process the request. The first StreamingRecognizeRequest message must contain a streaming_config message.

bytes audio_content

The audio data to be recognized. Sequential chunks of audio data are sent in sequential StreamingRecognizeRequest messages. The first StreamingRecognizeRequest message must not contain audio data and all subsequent StreamingRecognizeRequest messages must contain audio data. The audio bytes must be encoded as specified in RecognitionConfig.

nvidia.riva.RequestId id

The ID to be associated with the request. If provided, this will be returned in the corresponding responses.

message StreamingRecognizeResponse
StreamingRecognitionResult results(repeated)

This repeated list contains the latest transcript(s) corresponding to audio currently being processed. Currently one result is returned, where each result can have multiple alternatives

nvidia.riva.RequestId id

The ID associated with the request

message WordInfo

Word-specific information for recognized words.

int32 start_time

Time offset relative to the beginning of the audio in ms and corresponding to the start of the spoken word. This field is only set if enable_word_time_offsets=true and only in the top hypothesis.

int32 end_time

Time offset relative to the beginning of the audio in ms and corresponding to the end of the spoken word. This field is only set if enable_word_time_offsets=true and only in the top hypothesis.

string word

The word corresponding to this set of information.

float confidence

The confidence estimate. A higher number indicates an estimated greater likelihood that the recognized word is correct. This field is set only for a non-streaming result or, for a streaming result where is_final=true. This field is not guaranteed to be accurate and users should not rely on it to be always provided. Although confidence can currently be roughly interpreted as a natural-log probability, the estimate computation varies with difference configurations, and is subject to change. The default of 0.0 is a sentinel value indicating confidence was not set.

int32 speaker_tag

Output only. A distinct integer value is assigned for every speaker within the audio. This field specifies which one of those speakers was detected to have spoken this word. Value ranges from ‘1’ to diarization_speaker_count. speaker_tag is set if enable_speaker_diarization = ‘true’ and only in the top alternative.