Embedding Models

This page provides detailed technical specifications for the embedding model family supported by NeMo Customizer. For information about supported features and capabilities, refer to Tested Models.

Llama Nemotron Embedding 1B v2

Model Entity Configuration

Create a Model Entity for this embedding model:

Training Options

• LoRA (merged): 1x 80GB GPU, tensor parallel size 1
• Full SFT: 1x 80GB GPU, tensor parallel size 1

Resource Requirements

• Minimum GPU Memory: 80GB
• Recommended GPU: A100
• Training Time: Varies based on dataset size and epochs

Hyperparameter and Data Recommendations

This fine-tuning recipe supports full fine-tuning, updating all 1 billion parameters, and requires careful hyperparameter and data selection to prevent overfitting.

The following table provides conservative hyperparameter defaults specifically optimized to prevent overfitting for embedding models:

NVIDIA recommends evaluating fine-tuned embedding models against the baseline to detect overfitting and potential performance degradation.

Deployment Configuration

• Full SFT and LoRA (merged):
• NIM Image: nvcr.io/nim/nvidia/llama-nemotron-embed-1b-v2:1.13.0
• GPU Count: 1x 80GB

Deployment and Inference

This model supports inference deployment through NVIDIA Inference Microservices (NIM). After customization, access your model through the Inference Gateway:

1. Deploy the model: Create a ModelDeploymentConfig and ModelDeployment to deploy your fine-tuned model. See about for details.
2. Access through Inference Gateway: The Inference Gateway provides unified access to all deployed models via three routing patterns:

• Model Entity routing: /v2/workspaces/{workspace}/inference/gateway/model/{name}/-/v1/embeddings
• Provider routing: /v2/workspaces/{workspace}/inference/gateway/provider/{deployment}/-/v1/embeddings

1
import os
2
from nemo_platform import NeMoPlatform
3
4
client = NeMoPlatform(
5
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
6
    workspace="default",
7
)
8
9
# Get pre-configured OpenAI client for inference
10
oai_client = client.models.get_openai_client()

Example Usage

After fine-tuning and deployment, you can use the model for embedding tasks:

1
import os
2
from nemo_platform import NeMoPlatform
3
4
client = NeMoPlatform(
5
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
6
    workspace="default",
7
)
8
9
MODEL_NAME = "llama-nemotron-embed-1b-v2"
10
DEPLOYMENT_NAME = "my-embedding-deployment"
11
12
# Call embeddings via Inference Gateway
13
response = client.inference.gateway.provider.post(
14
    "v1/embeddings",
15
    name=DEPLOYMENT_NAME,
16
    workspace="default",
17
    body={
18
        "model": MODEL_NAME,
19
        "input": ["What is the capital of France?"],
20
        "input_type": "query",
21
    },
22
)
23
24
embedding = response["data"][0]["embedding"]
25
print(f"Embedding dimension: {len(embedding)}")

For detailed fine-tuning instructions, refer to the Embedding Customization tutorial.

For more information about formatting training datasets for the embedding model, refer to Dataset Format Requirements.