API Usage Examples (OpenAPI) for NVIDIA NeMo Retriever Embedding NIM#

Use the examples in this documentation to help you get started using the API for NVIDIA NeMo Retriever Embedding NIM.

Endpoints#

The runtime exposes the following endpoints. For the full API reference, refer to API Reference (OpenAPI) for NVIDIA NeMo Retriever Embedding NIM.

Method	Path	Description
`POST`	`/v1/embeddings`	Generate embeddings.
`GET`	`/v1/models`	List served models.
`GET`	`/v1/health/ready`	Check readiness.
`GET`	`/v1/health/live`	Check liveness.
`GET`	`/v1/metrics`	Return Prometheus metrics.
`GET`	`/v1/metadata`	Return runtime and model metadata.
`GET`	`/v1/manifest`	Return the model manifest.
`GET`	`/v1/license`	Return license information.
`GET`	`/v1/version`	Return runtime API version information.

The /v1 path prefix is the API version, not the model version.

Set the host and service port before you run the examples in this documentation:

export HOSTNAME=localhost
export SERVICE_PORT=8000

Health Checks#

The following example queries the health/ready endpoint to see if the service is ready to receive requests.

cURL Request

curl "http://${HOSTNAME}:${SERVICE_PORT}/v1/health/ready" \
-H 'Accept: application/json'

Response

{
  "object": "health.response",
  "message": "ready",
  "ready": true
}

The following example queries the health/live endpoint to see if the service is up and running.

cURL Request

curl "http://${HOSTNAME}:${SERVICE_PORT}/v1/health/live" \
-H 'Accept: application/json'

Response

{
  "object": "health.response",
  "message": "live",
  "live": true
}

Overview#

Maximum Token Length#

Every model has a maximum token length. The models section lists the maximum token lengths of the supported models. Use the truncate field to control how the runtime handles input that is longer than the served profile supports. For the field definition and the TruncateValue schema, refer to API Reference (OpenAPI) for NVIDIA NeMo Retriever Embedding NIM.

For Nemotron 3.5 Embed 1B, truncate accepts END and NONE, and the default is END. Omitting the field or setting it to null is the same as END. END truncates the end of the text to fit the deployment sequence limit. NONE rejects inputs that exceed the applicable limits. The server rejects START with HTTP 400 before inference, for both text-only and image requests. For image inputs, the complete image block and the passage prefix must fit within the limit. If they don’t fit, the request fails, even with END.

Passage and Query Modes#

Models such as nvidia/nemotron-3-embed-1b, NV-Embed-QA, and E5 operate in passage or query mode, and thus require the input_type parameter. passage is used when generating embeddings during indexing. query is used when generating embeddings during querying. It is very important to use the correct input_type. Failure to do so results in large drops in retrieval accuracy.

Since the OpenAI API does not accept input_type as a parameter, it is possible to add the -query or -passage suffix to the model parameter like NV-Embed-QA-query and not use the input_type field at all for OpenAI API compliance.

For example, the following two requests are identical.

With the input_type parameter:

curl -X "POST" \
  "http://${HOSTNAME}:${SERVICE_PORT}/v1/embeddings" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "input": ["What is the population of Pittsburgh?"],
    "model": "nvidia/nv-embedqa-e5-v5",
    "input_type": "query",
    "modality": "text"
}'

Without the input_type parameter with the -query (or -passage) in the model name:

curl -X "POST" \
  "http://${HOSTNAME}:${SERVICE_PORT}/v1/embeddings" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "input": ["What is the population of Pittsburgh?"],
    "model": "nvidia/nv-embedqa-e5-v5-query",
    "modality": "text"
}'

Embeddings#

Use POST /v1/embeddings to generate embeddings. Supported input modalities are model-specific.

Request Body#

Field	Type	Required	Description
`model`	string	Yes	Served model ID returned by `/v1/models`, such as `nvidia/Nemotron-3.5-Embed-1B-BF16` or `nvidia/Nemotron-3.5-Embed-1B-NVFP4`.
`input`	string or array of strings	Yes	Text, image data URL, or text with an image data URL, depending on the model.
`input_type`	string	No	Use `query` for text queries and `passage` for documents. For `nvidia/llama-nemotron-embed-vl-1b-v2`, this field is required, and image inputs require `passage`.
`modality`	string or array of strings	No	One of `text`, `image`, or `text_image`, depending on the model. If an array is provided, its length must match `input`, or it must contain one batch-wide value.
`embedding_type`	string	No	Embedding output type. Default is `float`. For Nemotron 3.5 Embed 1B, release 2.4 supports `float`, `int8`, and `uint8`. For other models, refer to the support matrix.
`encoding_format`	string	No	Output encoding: `float` or `base64`. Default is `float`. This field is independent of `embedding_type`; base64 does not enable an unsupported embedding type.
`dimensions`	integer	No	Dynamic embedding size. Supported values are listed in How to Specify Dynamic Embedding Sizes.
`longest_edge`	integer	No	For Nemotron 3.5 Embed 1B image inputs, the resize limit for the longest edge of each image. Accepts integers from `256` through `1540`, inclusive. If omitted, the model default of `1120` is used.
`image_longest_edge`	integer	No	Alias for `longest_edge`. Don’t provide both fields in the same request.
`truncate`	string	No	How to handle input that exceeds the maximum token length. For Nemotron 3.5 Embed 1B, one of `END` (default) or `NONE`; `START` is rejected. For other models, one of `START`, `END`, or `NONE`. For details, refer to Maximum Token Length.
`user`	string	No	Optional caller-provided user identifier.

Response Body#

The response contains an object with:

Field	Type	Description
`object`	string	Always `list`.
`data`	array	One embedding object per input item.
`model`	string	Served model ID.
`usage`	object	Token usage information.

Each item in data contains:

Field	Type	Description
`object`	string	Always `embedding`.
`index`	integer	Input index.
`embedding`	array or string	Embedding vector or, for supporting models, compressed embedding values. With `encoding_format: "base64"`, this field is a base64-encoded string.

How to Specify Dynamic Embedding Sizes#

To reduce the storage cost of the returned embeddings, some models support dynamic embedding sizes by using Matryoshka Representation Learning. To produce a lower-dimensional embedding representation of your text, you can use the optional dimensions API parameter. For the full list of supported models, refer to the support matrix.

The dimensions request parameter is model-specific. For nvidia/nemotron-3-embed-1b, omit dimensions or set it to 2048 to receive the native 2048-dimensional embedding output. Reduced dimensions are not supported.

Model	Native Dimension	Supported `dimensions` Values	Reduced Dimensions
`nvidia/nemotron-3-embed-1b`	2048	Omitted or `2048`	Not supported
`nvidia/Nemotron-3.5-Embed-1B-BF16`	2048	Any integer from `128` through `2048`, inclusive	Supported
`nvidia/Nemotron-3.5-Embed-1B-NVFP4`	2048	Any integer from `128` through `2048`, inclusive	Supported

Both Nemotron 3.5 models accept values such as 129, 777, and 2047; they do not require a fixed set of dimensions. The server keeps the first dimensions values of the embedding, then applies L2 normalization to the reduced vector. Omitting dimensions or setting it to null returns the native 2048-dimensional vector. These constraints also apply when the deployment uses a custom served model name. Values below 128 or above 2048 return HTTP 422. Use embedding_type: "float" when testing reduced-dimensional retrieval accuracy.

The model nvidia/llama-nemotron-embed-vl-1b-v2 accepts the following fixed dimensions:

128
256
384
512
768
1024
1536
2048

Important

When you use the dimensions API parameter, use the same value for dimensions in concurrent requests to preserve dynamic batching efficiency.

Limitations#

The following are limitations when you specify dimensions:

The model nvidia/nemotron-3-embed-1b supports only the native 2048-dimensional embedding output. Requests with unsupported values such as 128, 512, or 1024 return HTTP 400 with dimensions must be one of 2048.
For models other than the two Nemotron 3.5 models listed above, the dimensions parameter cannot be used in combination with the embedding_type parameter. Nemotron 3.5 accepts dimensions together with embedding_type: "float". It cannot combine dimensions with a compressed embedding_type (int8 or uint8). These are alternative methods for reducing the memory footprint of embeddings and have different performance trade-offs.
If dimensions is omitted, the model returns its native embedding dimension. The default embedding_type is float.

How to Specify Embedding Type#

The /v1/embeddings endpoint contains an optional field named embedding_type that selects the output representation. Supported values depend on the model. For the full list of models and their embedding types, refer to the support matrix.

Important

In release 2.4, Nemotron 3.5 Embed 1B supports three embedding_type values: float, int8, and uint8. You can also omit embedding_type; the default is float. The values binary and ubinary are not supported in release 2.4. The server rejects them before inference with HTTP 422, including requests that use a custom served model name. This output support applies to text and image inputs with both BF16 and NVFP4 model profiles.

The embedding_type field controls output quantization, not model-weight or inference precision such as BF16 or NVFP4. The separate encoding_format field controls serialization. Nemotron 3.5 Embed supports both encoding_format: "float" and encoding_format: "base64" independently of the selected embedding type. Base64 encoding does not enable an unsupported embedding_type.

For models that support the selected embedding type:

float — Use when you need maximum accuracy, and storage and memory constraints are not a concern. float is the default embedding type.
int8, uint8 — Recommended for most production deployments. Provides a balance of compression and accuracy.
binary, ubinary — Use for very large-scale systems, where maximum compression is critical, and you can accept some reduced accuracy.

You can specify embedding_type to potentially reduce memory and storage costs, and make it easier to scale vector databases to large datasets. Models that support compressed embedding types are optimized to minimize accuracy loss when you use these representations.

The model nvidia/nemotron-3-embed-1b supports float, int8, uint8, binary, and ubinary embeddings.

Embedding Type	Potential Memory Savings	Size per Dimension	Data Type Returned
`float`	1x	4 bytes	`float32`
`int8`	4x	1 byte	`int8`
`uint8`	4x	1 byte	`uint8`
`binary`	32x	1 bit	`int8`
`ubinary`	32x	1 bit	`uint8`

The following table shows the returned JSON values and embedding length for nvidia/llama-nemotron-embed-vl-1b-v2 with encoding_format: "float" and native dimensions. For Nemotron 3.5 Embed, only the float, int8, and uint8 rows apply in release 2.4.

Embedding Type	Returned JSON Values	Length
`float`	floating-point numbers	2048
`int8`	signed integers	2048
`uint8`	unsigned integers	2048
`binary`	packed signed integers	256
`ubinary`	packed unsigned integers	256

int8 Embeddings#

This section applies only to models that list int8 or uint8 support in the support matrix.

int8 embeddings reduce memory usage by a factor of four compared to float embeddings, while maintaining high retrieval accuracy. The conversion process maps each floating-point value in the original vector to an 8-bit integer, signed (int8) or unsigned (uint8).

For Nemotron 3.5 Embed BF16, uint8 values have an offset of 128 relative to int8 values. Before dot-product scoring, convert query and document codes to a wider numeric type, such as int32 or float32. For uint8, subtract 128 from both vectors after conversion. Do not score the raw unsigned values directly. Use the same integer encoding for queries and stored documents. When changing the encoding, rebuild the document index and switch query encoding together.

binary Embeddings#

This section applies only to models that list binary or ubinary support in the support matrix.

binary embeddings reduce storage requirements by a factor of 32 and can accelerate search speeds. This makes them ideal for systems that handle large datasets or require very low latency.

The conversion process maps each floating-point value in the original vector into a single bit (0 or 1), and then packs the bits into 8-bit bytes. For example, a 1024-dimension float embedding is transformed into 1024 bits, which are then represented as 128 int8 values (binary) or uint8 values (ubinary).

Warning

Binary embeddings require hamming distance for similarity calculations, not cosine similarity or dot product. If you specify binary for your embedding type, you might need to change your search implementation. Verify that your vector database supports the compressed type that you select.

Limitations#

For Nemotron 3.5 Embed, use float, int8, or uint8, or omit embedding_type to use float. The binary and ubinary types are not supported in release 2.4, and requests that set them return HTTP 422. The following limitations apply to compressed outputs from models that support them:

For models other than nvidia/Nemotron-3.5-Embed-1B-BF16 and nvidia/Nemotron-3.5-Embed-1B-NVFP4, the embedding_type parameter cannot be used in combination with the dimensions parameter. Both Nemotron 3.5 models accept embedding_type: "float" together with dimensions. Compressed embedding_type values (int8, uint8) cannot be combined with the dimensions parameter. These are alternative methods for reducing the memory footprint of embeddings and have different performance trade-offs.
Compressed embedding types (int8, uint8, binary, ubinary) can result in a loss of accuracy compared to float embeddings. The impact on retrieval accuracy varies depending on the embedding type and model.
Not all vector databases support compressed embedding types. Verify that your vector database supports int8 or binary embeddings before using these types in production.

How to Specify Modality#

The /v1/embeddings endpoint contains a modality field to support text, image, and mixed (text+image) input types. Supported modalities are model-specific. The model nvidia/nemotron-3-embed-1b supports text only. For models that support multimodal input, the following are the valid values for modality:

"text"
"image"
"text_image"

For image input, image data URLs are accepted in image/png or image/jpeg format, such as data:image/png;base64,.... You can also specify an image tag that wraps a data URL, such as <img src="data:image/png;base64,..."/>. Each input is validated against model limits.

Image Input Limitations#

The following limitations apply to image inputs:

The model nvidia/nemotron-3-embed-1b does not support image inputs.
For nvidia/llama-nemotron-embed-vl-1b-v2, images are document inputs. Use input_type: "passage". Images with input_type: "query" are rejected.
Decoded inline-image payloads are limited to 5 MiB by default. Set NIM_SERVER_MAX_IMAGE_BYTES to configure this limit. Encoded data URLs must also fit the OpenAPI input limit of 16,777,216 characters.
Image dimensions are limited to 8192 x 16384 or 16384 x 8192.

Specify Modality Explicitly#

If you specify modality explicitly, you ensure that each input is processed exactly as you intend. The following are the two ways to specify modality:

Specify a single string for single input. For example: "modality": "text".
Specify an array that matches the length of input for batched requests. For example: "modality": ["text", "image", "text_image"].

Let the Server Infer Modality#

If you omit modality, the server infers the modality for each input by using the following process:

If the input is a valid data URL starting with data:image/, the modality is inferred as image.
If the input contains both text and an embedded image data URL, such as caption text data:image/png;base64,..., the modality is inferred as text_image.
Otherwise the modality is inferred as text.

In some cases, such as malformed data URLs, or text that unintentionally mimics a data URL pattern, there is a risk that the server infers modality incorrectly.

How to Tune Dynamic Batching#

Dynamic batching is a feature that allows the NIM to group one or more requests into a single batch, which can improve throughput under certain conditions, for example when serving many requests with small payloads. This feature is enabled by default and can be tuned by setting the NIM_SERVER_MAX_WAIT_MS environment variable, in milliseconds. The default is 0 in latency mode and 10 in throughput mode (NIM_PERFORMANCE_MODE=1). The earlier name NIM_MAX_WAIT_MS isn’t supported in release 2.4. For details, refer to Environment Variables for NVIDIA NeMo Retriever Embedding NIM.

Example to List Models#

To list the available models, use the following code.

cURL Request

curl "http://${HOSTNAME}:${SERVICE_PORT}/v1/models" \
-H 'Accept: application/json'

Response

{
  "object": "list",
  "data": [
    {
      "id": "nvidia/Nemotron-3.5-Embed-1B-BF16",
      "created": 1779146207,
      "object": "model",
      "owned_by": "nvidia"
    }
  ]
}

The created value is the Unix timestamp of server startup, so it differs for each deployment.

Example to Generate a Text Query Embedding#

The following example requests a float embedding for a text query. You can omit embedding_type to use the same default, float. To receive the float vector as a base64-encoded string, change only encoding_format to base64.

These examples use nvidia/Nemotron-3.5-Embed-1B-BF16. On NVIDIA GB200, NVIDIA B300, and NVIDIA RTX PRO 6000 Blackwell Server Edition, the NIM serves nvidia/Nemotron-3.5-Embed-1B-NVFP4 by default. To confirm the model ID that your deployment serves, query /v1/models.

cURL Request

curl -X "POST" \
  "http://${HOSTNAME}:${SERVICE_PORT}/v1/embeddings" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "input": ["What is NVIDIA?"],
    "model": "nvidia/Nemotron-3.5-Embed-1B-BF16",
    "input_type": "query",
    "modality": "text",
    "embedding_type": "float",
    "encoding_format": "float"
}'

Response

The following response illustrates the format and is shortened for readability. Embedding values and token counts depend on the input and model profile.

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "index": 0,
      "embedding": [0.012519836, -0.0126571655, 0.0101623535]
    }
  ],
  "model": "nvidia/Nemotron-3.5-Embed-1B-BF16",
  "usage": {
    "prompt_tokens": 7,
    "total_tokens": 7
  }
}

Example to Generate an Image Document Embedding#

For nvidia/llama-nemotron-embed-vl-1b-v2, use input_type: "passage" and modality: "image" for image documents. The following example creates a valid PNG data URL, writes the request body to payload-image.json, and sends the request.

cURL Request

python3 - <<'PY'
import base64
import json
import struct
import zlib

width = height = 224
rows = []
for y in range(height):
    row = bytearray([0])
    for x in range(width):
        row.extend((x % 256, y % 256, 128))
    rows.append(bytes(row))

def chunk(kind, data):
    return (
        struct.pack(">I", len(data))
        + kind
        + data
        + struct.pack(">I", zlib.crc32(kind + data) & 0xFFFFFFFF)
    )

png = (
    b"\x89PNG\r\n\x1a\n"
    + chunk(b"IHDR", struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0))
    + chunk(b"IDAT", zlib.compress(b"".join(rows), 9))
    + chunk(b"IEND", b"")
)

payload = {
    "input": ["data:image/png;base64," + base64.b64encode(png).decode()],
    "model": "nvidia/llama-nemotron-embed-vl-1b-v2",
    "input_type": "passage",
    "modality": "image",
    "embedding_type": "float",
    "encoding_format": "float",
}

with open("payload-image.json", "w", encoding="utf-8") as f:
    json.dump(payload, f)
PY

curl -X "POST" \
  "http://${HOSTNAME}:${SERVICE_PORT}/v1/embeddings" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d @payload-image.json

Response

The following response is shortened for readability.

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "index": 0,
      "embedding": [-0.0047950745, 0.017486572, 0.023162842]
    }
  ],
  "model": "nvidia/llama-nemotron-embed-vl-1b-v2",
  "usage": {
    "prompt_tokens": 266,
    "total_tokens": 266
  }
}

Example to Generate a Compressed Embedding#

The following example generates an int8 embedding by specifying embedding_type. To request unsigned integer output, change embedding_type to uint8. Keep encoding_format unchanged for a JSON array.

cURL Request

curl -X "POST" \
  "http://${HOSTNAME}:${SERVICE_PORT}/v1/embeddings" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "input": ["What is NVIDIA?"],
    "model": "nvidia/Nemotron-3.5-Embed-1B-BF16",
    "input_type": "query",
    "modality": "text",
    "embedding_type": "int8",
    "encoding_format": "float"
}'

Response

The following response is illustrative and shortened for readability. Embedding values and token counts depend on the input and deployed model.

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "index": 0,
      "embedding": [1, -1, 1, 0, 5]
    }
  ],
  "model": "nvidia/Nemotron-3.5-Embed-1B-BF16",
  "usage": {
    "prompt_tokens": 7,
    "total_tokens": 7
  }
}

Error Responses#

With the default error format, HTTP errors are returned as JSON objects with object: "error", a message, and a type.

The following examples show common validation failures:

Case	HTTP Status	Message
Empty input array	`400`	`input must not be empty`
Blank input string	`400`	`input[0] must not be blank or empty`
Unknown model	`404`	`model 'wrong-model' not found; available: 'nvidia/Nemotron-3.5-Embed-1B-BF16'`
Invalid modality	`422`	`modality must be one of 'text', 'image', 'text_image'; got 'bogus'`
Missing `input_type` for an asymmetric model	`400`	`'input_type' parameter is required for asymmetric models`
Image with `input_type: "query"`	`400`	`input_type="query" is not supported with images; the VLM model processes images as passages only. Use input_type="passage".`
Invalid `encoding_format`	`422`	`encoding_format must be one of 'float', 'base64'; got 'bad'`
Invalid `embedding_type`	`422`	`Input should be 'float', 'binary', 'ubinary', 'int8' or 'uint8'`
`embedding_type: "binary"` or `"ubinary"` for Nemotron 3.5 Embed 1B	`422`	`embedding_type must be one of 'float', 'int8', 'uint8' for Nemotron 3.5 Embed`
Unsupported `dimensions` for `nvidia/nemotron-3-embed-1b`	`400`	`dimensions must be one of 2048`
`truncate: "START"` for Nemotron 3.5 Embed 1B	`400`	`Input should be 'END' or 'NONE'; got 'START'`
`longest_edge` outside `256` through `1540`	`400`	`longest_edge must be an integer from 256 through 1540`

API Usage Examples (OpenAPI) for NVIDIA NeMo Retriever Embedding NIM#

Endpoints#

Health Checks#

Overview#

Maximum Token Length#

Passage and Query Modes#

Embeddings#

Request Body#

Response Body#

How to Specify Dynamic Embedding Sizes#

Limitations#

How to Specify Embedding Type#

int8 Embeddings#

binary Embeddings#

Limitations#

How to Specify Modality#

Image Input Limitations#

Specify Modality Explicitly#

Let the Server Infer Modality#

How to Tune Dynamic Batching#

Example to List Models#

Example to Generate a Text Query Embedding#

Example to Generate an Image Document Embedding#

Example to Generate a Compressed Embedding#

Error Responses#

Related Topics#