Use the API (OpenAI) for NVIDIA NeMo Retriever Embedding NIM#
Use the examples in this documentation to help you get started using the API for NVIDIA NeMo Retriever Embedding NIM.
Endpoints#
The runtime exposes the following endpoints. For the full API reference, refer to API Reference (OpenAI).
Method |
Path |
Description |
|---|---|---|
|
|
Generate embeddings. |
|
|
List served models. |
|
|
Check readiness. |
|
|
Check liveness. |
|
|
Return Prometheus metrics. |
|
|
Return runtime and model metadata. |
|
|
Return the model manifest. |
|
|
Return license information. |
|
|
Return runtime API version information. |
The /v1 path prefix is the API version, not the model version.
Set the host and service port before you run the examples in this documentation:
export HOSTNAME=localhost
export SERVICE_PORT=8000
Health Checks#
The following example queries the health/ready endpoint to see if the service is ready to receive requests.
cURL Request
curl "http://${HOSTNAME}:${SERVICE_PORT}/v1/health/ready" \
-H 'Accept: application/json'
Response
{
"object": "health.response",
"message": "ready",
"ready": true
}
The following example queries the health/live endpoint to see if the service is up and running.
cURL Request
curl "http://${HOSTNAME}:${SERVICE_PORT}/v1/health/live" \
-H 'Accept: application/json'
Response
{
"object": "health.response",
"message": "live",
"live": true
}
Overview#
Maximum Token Length#
Every model has a maximum token length. The models section lists the maximum token lengths of the supported models.
Use the truncate field to control how the runtime handles input that is longer than the served profile supports. For the field definition and the TruncateValue schema, refer to API Reference (OpenAI).
Passage and Query Modes#
Models such as nvidia/nemotron-3-embed-1b, NV-Embed-QA, and E5 operate in passage or query mode, and thus require the input_type parameter. passage is used when generating
embeddings during indexing. query is used when generating embeddings during querying. It is very important to use the correct input_type.
Failure to do so will result in large drops in retrieval accuracy.
Since the OpenAI API does not accept input_type as a parameter, it is possible to add the -query or -passage suffix to the model parameter
like NV-Embed-QA-query and not use the input_type field at all for OpenAI API compliance.
For example, the following two requests are identical.
With the input_type parameter:
curl -X "POST" \
"http://${HOSTNAME}:${SERVICE_PORT}/v1/embeddings" \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"input": ["What is the population of Pittsburgh?"],
"model": "nvidia/nv-embedqa-e5-v5",
"input_type": "query",
"modality": "text"
}'
Without the input_type parameter with the -query (or -passage) in the model name:
curl -X "POST" \
"http://${HOSTNAME}:${SERVICE_PORT}/v1/embeddings" \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"input": ["What is the population of Pittsburgh?"],
"model": "nvidia/nv-embedqa-e5-v5-query",
"modality": "text"
}'
The GTE and GTR models do not accept the input_type parameter, since both the -query and -passage input types are processed in the same way.
Embeddings#
Use POST /v1/embeddings to generate embeddings. Supported input modalities are model-specific.
Request Body#
Field |
Type |
Required |
Description |
|---|---|---|---|
|
string |
Yes |
Model ID, such as |
|
string or array of strings |
Yes |
Text, image data URL, or text with an image data URL, depending on the model. |
|
string |
No |
Use |
|
string or array of strings |
No |
One of |
|
string |
No |
Embedding type. Supported values depend on the model. Default is |
|
string |
No |
One of |
|
integer |
No |
Dynamic embedding size. Supported values are listed in How to Specify Dynamic Embedding Sizes. |
|
string |
No |
One of |
|
string |
No |
Optional caller-provided user identifier. |
Response Body#
The response contains an object with:
Field |
Type |
Description |
|---|---|---|
|
string |
Always |
|
array |
One embedding object per input item. |
|
string |
Served model ID. |
|
object |
Token usage information. |
Each item in data contains:
Field |
Type |
Description |
|---|---|---|
|
string |
Always |
|
integer |
Input index. |
|
array |
Embedding vector or packed compressed embedding values. |
How to Specify Dynamic Embedding Sizes#
To reduce the storage cost of the returned embeddings,
some models support dynamic embedding sizes by using Matryoshka Representation Learning.
To produce a lower-dimensional embedding representation of your text,
you can use the optional dimensions API parameter.
For the full list of supported models, refer to the support matrix.
The dimensions request parameter is model-specific. For nvidia/nemotron-3-embed-1b, omit dimensions or set it to 2048 to receive the native 2048-dimensional embedding output. Reduced dimensions are not supported.
Model |
Native Dimension |
Supported |
Reduced Dimensions |
|---|---|---|---|
|
2048 |
Omitted or |
Not supported |
The VLM Embed runtime accepts the following dimensions:
128256384512768102415362048
Important
When you use the dimensions API parameter, use the same value for dimensions in concurrent requests to preserve dynamic batching efficiency.
Limitations#
The following are limitations when you specify dimensions:
The model
nvidia/nemotron-3-embed-1bsupports only the native 2048-dimensional embedding output. Requests with unsupported values such as128,512, or1024return HTTP400withdimensions must be one of 2048.The
dimensionsparameter cannot be used in combination with theembedding_typeparameter. These are alternative methods for reducing the memory footprint of embeddings and have different performance trade-offs.If
dimensionsis omitted, the model returns its native embedding dimension. The defaultembedding_typeisfloat.
How to Specify Embedding Type#
The /v1/embeddings endpoint contains an optional field named embedding_type
that supports the following types of embeddings.
For the full list of models and their embedding types, refer to the support matrix.
float — Use when you need maximum accuracy, and storage and memory constraints are not a concern.
floatis the default embedding type.int8, uint8 — Recommended for most production deployments. Provides a balance of compression and accuracy.
binary, ubinary — Use for very large-scale systems, where maximum compression is critical, and you can accept some reduced accuracy.
You can specify embedding_type to potentially reduce memory and storage costs,
and make it easier to scale vector databases to large datasets.
Models that support compressed embedding types are optimized to minimize accuracy loss when you use these representations.
The model nvidia/nemotron-3-embed-1b supports float, int8, uint8, binary, and ubinary embeddings.
Embedding Type |
Potential Memory Savings |
Size per Dimension |
Data Type Returned |
|---|---|---|---|
|
1x |
4 bytes |
|
|
4x |
1 byte |
|
|
4x |
1 byte |
|
|
32x |
1 bit |
|
|
32x |
1 bit |
|
The following table shows the returned JSON values and the resulting embedding length for the VLM Embed runtime.
Embedding Type |
Returned JSON Values |
Length for VLM Embed |
|---|---|---|
|
floating-point numbers |
2048 |
|
signed integers |
2048 |
|
unsigned integers |
2048 |
|
packed signed integers |
256 |
|
packed unsigned integers |
256 |
int8 Embeddings#
int8 embeddings reduce memory usage by a factor of four compared to float embeddings,
while maintaining high retrieval accuracy.
The conversion process maps each floating-point value in the original vector to an 8-bit integer, signed (int8) or unsigned (uint8).
binary Embeddings#
binary embeddings reduce storage requirements by a factor of 32 and can accelerate search speeds.
This makes them ideal for systems that handle large datasets or require very low latency.
The conversion process maps each floating-point value in the original vector into a single bit (0 or 1),
and then packs the bits into 8-bit bytes.
For example, a 1024-dimension float embedding is transformed into 1024 bits, which are then represented as 128 int8 values (binary) or uint8 values (ubinary).
Warning
Binary embeddings require hamming distance for similarity calculations, not cosine similarity or dot product.
If you specify binary for your embedding type, you might need to change your search implementation.
Verify that your vector database supports the compressed type that you select.
Limitations#
The following are limitations when you specify embedding_type:
The
embedding_typeparameter cannot be used in combination with thedimensionsparameter. These are alternative methods for reducing the memory footprint of embeddings and have different performance trade-offs.Compressed embedding types (
int8,uint8,binary,ubinary) can result in a loss of accuracy compared tofloatembeddings. The impact on retrieval accuracy varies depending on the embedding type and model.Not all vector databases support compressed embedding types. Verify that your vector database supports
int8or binary embeddings before using these types in production.
How to Specify Modality#
The /v1/embeddings endpoint contains a modality field to support text, image, and mixed (text+image) input types.
Supported modalities are model-specific. The model nvidia/nemotron-3-embed-1b supports text only.
For models that support multimodal input, the following are the valid values for modality:
"text""image""text_image"
For image input, image data URLs are accepted in image/png or image/jpeg format, such as data:image/png;base64,....
You can also specify an image tag that wraps a data URL, such as <img src="data:image/png;base64,..."/>.
Each input is validated against model limits.
Image Input Limitations#
The following limitations apply to image inputs:
The model
nvidia/nemotron-3-embed-1bdoes not support image inputs.For
nvidia/llama-nemotron-embed-vl-1b-v2, images are document inputs. Useinput_type: "passage". Images withinput_type: "query"are rejected.Decoded inline-image payloads are limited to 5 MiB by default. Set
NIM_SERVER_MAX_IMAGE_BYTESto configure this limit. Encoded data URLs must also fit the OpenAPIinputlimit of 16,777,216 characters.Image dimensions are limited to
8192 x 16384or16384 x 8192.
Specify Modality Explicitly#
If you specify modality explicitly, you ensure that each input is processed exactly as you intend.
The following are the two ways to specify modality:
Specify a single string for single input. For example:
"modality": "text".Specify an array that matches the length of input for batched requests. For example:
"modality": ["text", "image", "text_image"].
Let the Server Infer Modality#
If you omit modality, the server infers the modality for each input by using the following process:
If the input is a valid data URL starting with
data:image/, themodalityis inferred asimage.If the input contains both text and an embedded image data URL, such as
caption text data:image/png;base64,..., the modality is inferred astext_image.Otherwise the
modalityis inferred astext.
In some cases, such as malformed data URLs, or text that unintentionally mimics a data URL pattern, there is a risk that the server infers modality incorrectly.
How to Tune Dynamic Batching#
Dynamic batching is a feature that allows the NIM to group one or more requests into a single batch, which can improve throughput under certain conditions, for example when serving many requests with small payloads. This feature is enabled by default and can be tuned by setting the NIM_MAX_WAIT_MS environment variable. The default value is 10ms (milliseconds).
Example to List Models#
To list the available models, use the following code.
cURL Request
curl "http://${HOSTNAME}:${SERVICE_PORT}/v1/models" \
-H 'Accept: application/json'
Response
{
"object": "list",
"data": [
{
"id": "nvidia/nemotron-3-embed-1b",
"created": 1779146207,
"object": "model",
"owned_by": "nvidia"
}
]
}
The created value is the Unix timestamp of server startup, so it differs for each deployment.
Example to Generate a Text Query Embedding#
For models that require the input_type parameter, use input_type: "query" for text queries.
cURL Request
curl -X "POST" \
"http://${HOSTNAME}:${SERVICE_PORT}/v1/embeddings" \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"input": ["What is NVIDIA?"],
"model": "nvidia/nemotron-3-embed-1b",
"input_type": "query",
"modality": "text",
"embedding_type": "float",
"encoding_format": "float"
}'
Response
The following response is shortened for readability.
{
"object": "list",
"data": [
{
"object": "embedding",
"index": 0,
"embedding": [0.012519836, -0.0126571655, 0.0101623535]
}
],
"model": "nvidia/nemotron-3-embed-1b",
"usage": {
"prompt_tokens": 7,
"total_tokens": 7
}
}
Example to Generate Embeddings without Input Type#
For models that do not require the input_type parameter, such as GTE or GTR, use the following code to generate embeddings.
cURL Request
curl -X "POST" \
"http://${HOSTNAME}:${SERVICE_PORT}/v1/embeddings" \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"input": ["Hello world"],
"model": "nvidia/nv-embedqa-e5-v5",
"modality": "text"
}'
Response
{
"object": "list",
"data": [
{
"index": 0,
"embedding": [
0.0010356903076171875, -0.017669677734375,
// ...
-0.0178985595703125
],
"object": "embedding"
}
],
"model": "nvidia/nv-embedqa-e5-v5",
"usage": {
"prompt_tokens": 0,
"total_tokens": 0
}
}
Example to Generate an Image Document Embedding#
For nvidia/llama-nemotron-embed-vl-1b-v2, use input_type: "passage" and modality: "image" for image documents. The following example creates a valid PNG data URL, writes the request body to payload-image.json, and sends the request.
cURL Request
python3 - <<'PY'
import base64
import json
import struct
import zlib
width = height = 224
rows = []
for y in range(height):
row = bytearray([0])
for x in range(width):
row.extend((x % 256, y % 256, 128))
rows.append(bytes(row))
def chunk(kind, data):
return (
struct.pack(">I", len(data))
+ kind
+ data
+ struct.pack(">I", zlib.crc32(kind + data) & 0xFFFFFFFF)
)
png = (
b"\x89PNG\r\n\x1a\n"
+ chunk(b"IHDR", struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0))
+ chunk(b"IDAT", zlib.compress(b"".join(rows), 9))
+ chunk(b"IEND", b"")
)
payload = {
"input": ["data:image/png;base64," + base64.b64encode(png).decode()],
"model": "nvidia/llama-nemotron-embed-vl-1b-v2",
"input_type": "passage",
"modality": "image",
"embedding_type": "float",
"encoding_format": "float",
}
with open("payload-image.json", "w", encoding="utf-8") as f:
json.dump(payload, f)
PY
curl -X "POST" \
"http://${HOSTNAME}:${SERVICE_PORT}/v1/embeddings" \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d @payload-image.json
Response
The following response is shortened for readability.
{
"object": "list",
"data": [
{
"object": "embedding",
"index": 0,
"embedding": [-0.0047950745, 0.017486572, 0.023162842]
}
],
"model": "nvidia/llama-nemotron-embed-vl-1b-v2",
"usage": {
"prompt_tokens": 266,
"total_tokens": 266
}
}
Example to Generate a Compressed Embedding#
The following example generates an int8 embedding by specifying embedding_type.
cURL Request
curl -X "POST" \
"http://${HOSTNAME}:${SERVICE_PORT}/v1/embeddings" \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"input": ["What is NVIDIA?"],
"model": "nvidia/llama-nemotron-embed-vl-1b-v2",
"input_type": "query",
"modality": "text",
"embedding_type": "int8",
"encoding_format": "float"
}'
Response
The following response is shortened for readability.
{
"object": "list",
"data": [
{
"object": "embedding",
"index": 0,
"embedding": [1, -1, 1, 0, 5]
}
],
"model": "nvidia/llama-nemotron-embed-vl-1b-v2",
"usage": {
"prompt_tokens": 7,
"total_tokens": 7
}
}
Error Responses#
Errors are returned as JSON objects with object: "error", a message, and a type.
The following examples show common validation failures:
Case |
HTTP Status |
Message |
|---|---|---|
Empty input array |
|
|
Blank input string |
|
|
Unknown model |
|
|
Invalid modality |
|
|
Missing |
|
|
Image with |
|
|
Invalid |
|
|
Invalid |
|
|
Unsupported |
|
|