API Reference (gRPC) for NVIDIA NeMo Retriever Reranking NIM#
This documentation contains the gRPC reference for NVIDIA NeMo Retriever Reranking NIM.
gRPC Models#
The gRPC model names differ from the NIM model IDs that appear in the Support Matrix. The following table contains the mapping of the names.
Model ID |
gRPC Model Name |
|---|---|
nvidia/Nemotron-3.5-Rerank-8B-BF16 |
nvidia_nemotron_3_5_rerank_8b_bf16 |
Nemotron 3.5 Rerank supports unary gRPC inference through the KServe V2 protocol. To enable and publish the gRPC listener, refer to Launch the NIM.
The preceding table lists the default served names.
If you set a custom served model name with NIM_SERVER_SERVED_MODEL_NAME, the gRPC name is its lowercase form, with each character other than an ASCII letter or digit replaced by an underscore.
To confirm the served gRPC name, query the model metadata.
A served alias doesn’t change the loaded checkpoint.
Request Inputs#
Each Nemotron 3.5 Rerank request ranks one text query against N passages, where N is from 1 through 512.
The following table lists the request inputs for Nemotron 3.5 Rerank.
Input |
Shape |
Data Type |
Description |
Required |
|---|---|---|---|---|
|
[1, 1] or [N, 1] |
BYTES |
One UTF-8 query, or the same query repeated once per passage. Distinct queries in one request are rejected. |
Yes |
|
[N, 1] |
BYTES |
One UTF-8 text value per passage. |
Yes |
|
[N, 1] |
BYTES |
One UTF-8 PNG or JPEG base64 data URI per passage. Use an empty string for passages without an image. |
No |
One-dimensional shapes [1] and [N] are also accepted.
All other input shapes are rejected.
When you supply passage_image, it must contain exactly N values.
For an image-only passage, use an empty passage string and a nonempty image data URI.
Every passage must contain text, an image, or both.
Images use the checkpoint’s resize settings and the same payload limit as HTTP ranking.
Inputs can use protobuf contents.bytes_contents or KServe raw_input_contents.
For raw inputs, prefix each BYTES value with its length as a little-endian 32-bit unsigned integer.
Supply one raw buffer per input tensor and omit typed contents.
Don’t combine raw and typed contents.
Unknown or duplicate inputs, invalid UTF-8, and mismatched tensor counts are rejected before inference.
Request Parameters#
The following table lists the request parameters.
Parameter |
Data Type |
Description |
Valid Values |
Default |
Required |
|---|---|---|---|---|---|
|
String |
How to handle text that exceeds the maximum token length. |
|
|
No |
For Nemotron 3.5 Rerank, omitting truncate uses NONE, the same as HTTP ranking.
NONE rejects a query and passage pair that exceeds the configured sequence limit.
END truncates from the end.
The server rejects START before inference.
Use an empty model_version for the loaded checkpoint.
Streaming inference isn’t supported.
Response#
Nemotron 3.5 Rerank returns one-dimensional tensors in descending score order. Responses echo the request ID and the requested served model name. The following table lists the response outputs for Nemotron 3.5 Rerank.
Output |
Shape |
Data Type |
Description |
|---|---|---|---|
|
[N] |
INT32 |
Zero-based index in the original passage list. |
|
[N] |
FP32 |
Cross-encoder score for the passage at the corresponding index. |
|
[N] |
INT32 |
Effective query and passage sequence token count for that passage, including prompt and image tokens. |
Token counts follow the sorted indices.
Each count belongs to one passage; it isn’t the request’s total token usage repeated for every passage.
Responses use raw_output_contents with little-endian numeric encoding.
To receive all three outputs, omit the requested outputs, or request a subset by name.