> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server. # Embedding Model Customization [Run in Google Colab](https://colab.research.google.com/github/NVIDIA-NeMo/nemo-platform/blob/main/docs/customizer/tutorials/embedding-customization-job.ipynb) Fine-tune Nemotron 3 Embed 1B on domain retrieval data and score it against the base checkpoint with BEIR nDCG and Recall. This is the NeMo Platform mapping of the [Nemotron embed recipe](https://github.com/NVIDIA-NeMo/Nemotron/blob/main/docs/nemotron/embed/README.md). Pre-trained embeddings handle general retrieval well but underperform on specialized vocabulary, document structure, and query phrasing. Fine-tuning adapts the model to your corpus. Expect larger gains where the base model has the least prior exposure, such as legal, biomedical, and internal engineering documents, and smaller ones on corpora close to its pre-training mix. This notebook starts from NVIDIA's published NVDocs Stage 0 dump instead of generating Q\&A pairs, so the first job it runs is hard-negative mining. | Stage | Command | Notes | | ---------- | ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | 0 SDG | `nemo data-designer retrieval-generate` | Skipped here. Run it on your own corpus with [Generate retrieval training data](/documentation/design-synthetic-data/tutorials/retrieval-generate) | | 1 Prep | `nemo data-designer retrieval-prepare` with `enable_mining: true` | Writes `training.jsonl` and the frozen `eval_beir/` split | | 2 Finetune | Automodel `recipe: bi_encoder` | Alternate data: [SPECTER or triplet JSONL](/documentation/customizer-reference/tutorials/embedding-triplet-dataset) | | 3 Deploy | Retriever NIM 2.2.0 | Base and tuned endpoints | | 4 Eval | `nemo evaluator retrieve-eval` | nDCG\@k, Recall\@k, Precision\@k, MAP\@k | Deployment precedes evaluation because `retrieve-eval` scores Inference Gateway providers rather than filesets, so both models have to be served first. Export is a setting on the training job rather than a stage of its own. ## Dataset format Stage 1 writes contrastive rows as JSONL: ```json {"query": "What is machine learning?", "pos_doc": "Machine learning is a subset of AI...", "neg_doc": ["A related but incorrect passage..."]} ``` `neg_doc` is a non-empty list of hard negatives: passages that score close to the positive but do not answer the query. Mining fills it during Stage 1 against a fileset-backed encoder, and Automodel samples four of them per query (`train_n_passages: 5`). ## Prerequisites * A running NeMo Platform at `NMP_BASE_URL`, default `http://localhost:8080`. See the [Quickstart](/documentation/get-started). * The Python SDK: `pip install "nemo-platform[all]"`, or `make bootstrap` from the repo root. * An NGC API key for the `nvcr.io` NIM images. * GPU capacity: 80 GB for training, about 40 GB for mining and evaluation. Serving the base and tuned NIMs side by side takes two GPUs. * A fileset-backed encoder entity, created in the next section. Models that Inference Gateway discovers from a running endpoint have `fileset: null` and can be neither trained nor used as the mining encoder. ## 1. Initialize the SDK ```sh export NMP_BASE_URL= export NGC_API_KEY= ``` The `nemo` CLI reads `NMP_BASE_URL` too, and `--base-url` overrides it per command. The NGC key becomes a Platform secret so the cluster can pull NIM images: ```bash echo "$NGC_API_KEY" | nemo secrets create ngc-api-key --workspace default --from-file - ``` ```python import json import os import time import uuid from IPython.display import clear_output from nemo_platform import NeMoPlatform, ConflictError NMP_BASE_URL = os.environ.get("NMP_BASE_URL", "http://localhost:8080") client = NeMoPlatform(base_url=NMP_BASE_URL, workspace="default") print(f"Platform: {NMP_BASE_URL}") ``` ## 2. Register the trainable base (Nemotron 3 Embed 1B) Mining and training both read weights from a fileset, so register the Hugging Face checkpoint as a fileset and back a model entity with it. Equivalent CLI: ```bash nemo files filesets create nemotron-3-embed-1b --workspace default --purpose model --exist-ok \ --description "Nemotron 3 Embed 1B BF16 checkpoint" \ --storage '{"type":"huggingface","repo_id":"nvidia/Nemotron-3-Embed-1B-BF16","repo_type":"model"}' nemo models create nemotron-3-embed-1b --workspace default --exist-ok \ --fileset default/nemotron-3-embed-1b --trust-remote-code nemo models get nemotron-3-embed-1b --workspace default ``` The entity is ready once `nemo models get` reports a non-null `fileset` and a populated `spec`. For a gated repo, create the token first with `nemo secrets create hf-token --value "$HF_TOKEN"` and add `"token_secret":"hf-token"` to `--storage`. ```python from nemo_platform.types.files import HuggingfaceStorageConfigParam NGC_API_KEY = os.environ.get("NGC_API_KEY") if not NGC_API_KEY: raise ValueError("NGC_API_KEY is required to pull Retriever NIM images from nvcr.io") NGC_SECRET_NAME = "ngc-api-key" try: client.secrets.create(name=NGC_SECRET_NAME, workspace="default", value=NGC_API_KEY) print(f"Created secret: {NGC_SECRET_NAME}") except ConflictError: print(f"Secret '{NGC_SECRET_NAME}' already exists") HF_TOKEN = os.getenv("HF_TOKEN") HF_REPO_ID = "nvidia/Nemotron-3-Embed-1B-BF16" MODEL_NAME = "nemotron-3-embed-1b" storage_kwargs = {"type": "huggingface", "repo_id": HF_REPO_ID, "repo_type": "model"} if HF_TOKEN: try: client.secrets.create(name="hf-token", workspace="default", value=HF_TOKEN) except ConflictError: pass storage_kwargs["token_secret"] = "hf-token" storage = HuggingfaceStorageConfigParam(**storage_kwargs) try: client.files.filesets.create( workspace="default", name=MODEL_NAME, purpose="model", description="Nemotron 3 Embed 1B BF16 checkpoint", storage=storage, ) except ConflictError: print("Base model fileset already exists") try: base_model = client.models.create( workspace="default", name=MODEL_NAME, fileset=f"default/{MODEL_NAME}", trust_remote_code=True, ) print(f"Created Model Entity: {MODEL_NAME}") except ConflictError: base_model = client.models.update( workspace="default", name=MODEL_NAME, fileset=f"default/{MODEL_NAME}", trust_remote_code=True, ) print(f"Updated Model Entity: {MODEL_NAME}") print("Waiting for ModelSpec...") spec_start = time.time() while not getattr(base_model, "spec", None): if time.time() - spec_start > 180: raise TimeoutError("ModelSpec not populated") time.sleep(3) base_model = client.models.retrieve(workspace="default", name=MODEL_NAME) print(f"fileset={base_model.fileset} spec={base_model.spec}") if not base_model.fileset: raise RuntimeError("Entity has a null fileset; it cannot be trained. Recreate from the Hugging Face fileset.") ``` ## 3. Stage 1: mine hard negatives from the NVDocs dump [Retrieval-Synthetic-NVDocs-v1](https://huggingface.co/datasets/nvidia/Retrieval-Synthetic-NVDocs-v1) is NVIDIA's published Stage 0 output, so Stage 1 reads it directly and no LLM is called. Point `sdg_input` at the dump, turn on mining, and name the encoder entity from the previous section. A hard negative is a corpus passage that the current encoder ranks near the positive even though it does not answer the query. Mining embeds every training query and the whole corpus on GPU, then keeps the highest-scoring passages below `min_positive_score * hard_neg_margin`. Start with `hard_neg_margin: 0.95` and `hard_negatives_to_mine: 5`; Stage 2 uses four of those candidates with `train_n_passages: 5`. Raise the margin toward 1.0 for harder negatives, or lower it to 0.85–0.90 to reduce false negatives. Change it only if Stage 4 metrics plateau. See the Nemotron recipe's [hard-negative mining guidance](https://github.com/NVIDIA-NeMo/Nemotron/blob/main/docs/nemotron/embed/README.md#hard-negative-mining). To generate Q\&A pairs from your own corpus first, run [Generate retrieval training data](/documentation/design-synthetic-data/tutorials/retrieval-generate) and pass that job's artifacts fileset as `sdg_input`. Equivalent CLI: ```bash nemo data-designer retrieval-prepare --workspace default --spec '{ "sdg_input": "hf://nvidia/Retrieval-Synthetic-NVDocs-v1@1c0d1856f3fb595b2dda98d4b61061fa6d782d51/nv_pp_dd_sdg.json", "enable_mining": true, "model": "default/nemotron-3-embed-1b", "hard_negatives_to_mine": 5, "hard_neg_margin": 0.95, "mining": {"query_embedding_batch_size": 64, "document_embedding_batch_size": 64} }' nemo jobs watch --workspace default ``` ```python from nemo_data_designer_plugin.jobs.retrieval_spec import RetrievalPrepareJobConfig NVDOCS_SDG = "hf://nvidia/Retrieval-Synthetic-NVDocs-v1@1c0d1856f3fb595b2dda98d4b61061fa6d782d51/nv_pp_dd_sdg.json" prepare_spec = RetrievalPrepareJobConfig( sdg_input=NVDOCS_SDG, enable_mining=True, model=f"default/{MODEL_NAME}", hard_negatives_to_mine=5, hard_neg_margin=0.95, mining={ "query_embedding_batch_size": 64, "document_embedding_batch_size": 64, }, ) prepare_job = client.data_designer.retrieval_prepare(prepare_spec, workspace="default") print(f"Prepare job: {prepare_job.name}") prepare_job.wait_until_done() prepare_status = prepare_job.get_job_status() print(f"Stage 1 status: {prepare_status}") if prepare_status != "completed": raise RuntimeError(f"retrieval-prepare finished with status {prepare_status}") ``` Stage 1 writes `training.jsonl` and `eval_beir/` under its job result, addressed as `workspace/fileset#path`. In 0.6, Automodel `dataset.training` and `retrieve-eval` `dataset` take a fileset name without a path fragment, so the next cell copies both artifacts to the root of a fileset of its own. Release 0.7 resolves `workspace/fileset#path` directly and the copy is no longer needed. The copy also drops duplicate `(query-id, corpus-id)` rows from `eval_beir/qrels/test.tsv`, which the BEIR loader rejects. Freeze the resulting fileset and reuse it for every run. Scoring two runs on different splits measures data variance rather than model quality. Equivalent CLI, where `RESULT_PATH` is the path fragment of the `artifacts` result's `artifact_url`: ```bash PREPARE_JOB= RESULT_PATH=results//artifacts nemo jobs results list "$PREPARE_JOB" --workspace default --output json nemo files download "job-fileset-$PREPARE_JOB" --workspace default \ --remote-path "$RESULT_PATH/training.jsonl" \ -o ./artifacts/training.jsonl nemo files download "job-fileset-$PREPARE_JOB" --workspace default \ --remote-path "$RESULT_PATH/eval_beir" \ -o ./artifacts/ nemo files filesets create nvdocs-embed-artifacts --workspace default \ --purpose dataset --exist-ok nemo files upload ./artifacts/ nvdocs-embed-artifacts --workspace default nemo files list nvdocs-embed-artifacts --workspace default ``` Deduplicate `./artifacts/eval_beir/qrels/test.tsv` before the upload; the cell below does it inline. ```python import csv import tempfile from pathlib import Path results = client.jobs.results.list(prepare_job.name, workspace="default").data artifacts = next(row for row in results if row.name == "artifacts") ARTIFACTS_REF = artifacts.artifact_url print("Stage 1 artifacts:", ARTIFACTS_REF) TRAINING_FILESET = "nvdocs-embed-artifacts" fileset_ref, _, remote_prefix = ARTIFACTS_REF.partition("#") job_fileset = fileset_ref.split("/", 1)[-1] local_dir = Path(tempfile.mkdtemp(prefix="nvdocs-artifacts-")) client.files.download( remote_path=f"{remote_prefix}/training.jsonl", local_path=str(local_dir / "training.jsonl"), fileset=job_fileset, workspace="default", ) # eval_beir arrives under local_dir; a destination already named eval_beir nests as eval_beir/eval_beir client.files.download( remote_path=f"{remote_prefix}/eval_beir", local_path=str(local_dir), fileset=job_fileset, workspace="default", ) if not (local_dir / "eval_beir" / "corpus.jsonl").is_file(): raise RuntimeError(f"eval_beir/corpus.jsonl missing under {local_dir}") # the BEIR loader rejects duplicate judgments qrels_path = local_dir / "eval_beir" / "qrels" / "test.tsv" with qrels_path.open(encoding="utf-8", newline="") as stream: reader = csv.DictReader(stream, delimiter="\t") header = reader.fieldnames seen: set[tuple[str, str]] = set() kept: list[dict[str, str]] = [] for row in reader: key = (row["query-id"], row["corpus-id"]) if key not in seen: seen.add(key) kept.append(row) with qrels_path.open("w", encoding="utf-8", newline="") as stream: writer = csv.DictWriter(stream, fieldnames=header, delimiter="\t", lineterminator="\n") writer.writeheader() writer.writerows(kept) print(f"qrels judgments: {len(kept)}") try: client.files.filesets.create( workspace="default", name=TRAINING_FILESET, purpose="dataset", description="NVDocs mined retrieval artifacts", ) except ConflictError: pass client.files.upload( local_path=f"{local_dir}/", remote_path="", fileset=TRAINING_FILESET, workspace="default", ) TRAINING_REF = f"default/{TRAINING_FILESET}" print("dataset.training and retrieve-eval dataset:", TRAINING_REF) ``` ## 4. Stage 2: Automodel `bi_encoder` `model` and `dataset.training` are independent inputs. The entity supplies the encoder weights; the fileset supplies the contrastive rows, with `training.jsonl` at its root. To train on an existing triplet dataset instead, build that fileset in [Train on SPECTER or triplet JSONL](/documentation/customizer-reference/tutorials/embedding-triplet-dataset) and point `dataset.training` at it — SPECTER carries no `eval_beir`, so keep the NVDocs artifacts fileset for Stage 4. These values match the Nemotron embed recipe, except for epochs: the recipe's `3` is calibrated for its example dataset, and NVDocs-scale corpora train for 1–2. | Knob | Value | | ----------------------------------- | -------------------------------------------------------------------------- | | `training.recipe` | `bi_encoder` | | `training.finetuning_type` | `all_weights`, or `lora_merged` to merge adapters back into the checkpoint | | `training.retrieval.export.primary` | `hf` | | Sequence length | 512, the same at mine, train, and eval time | | LR / warmup / GBS / MBS | `1e-5` / `5` / `128` / `4` | | Epochs | `1` | `export.primary: hf` writes the Hugging Face checkpoint to the fileset root, where Retriever NIM 2.2.0 looks for it, and keeps ONNX under `alternates/onnx`. Unmerged `lora` is rejected for `bi_encoder`, since an embedding NIM serves a full checkpoint and cannot load a standalone adapter. Equivalent CLI: ```bash nemo customization automodel.jobs submit --workspace default --name embed-nvdocs --spec '{ "model": "default/nemotron-3-embed-1b", "dataset": {"training": "default/nvdocs-embed-artifacts"}, "training": { "training_type": "sft", "recipe": "bi_encoder", "finetuning_type": "all_weights", "max_seq_length": 512, "retrieval": { "query_max_length": 512, "passage_max_length": 512, "train_n_passages": 5, "export": {"primary": "hf"} } }, "schedule": {"epochs": 1}, "batch": {"global_batch_size": 128, "micro_batch_size": 4}, "optimizer": {"learning_rate": 1e-5, "warmup_steps": 5, "weight_decay": 0.01}, "parallelism": {"num_gpus_per_node": 1}, "output": {"name": "nemotron-3-embed-1b-tuned"} }' nemo jobs watch embed-nvdocs --workspace default ``` `training.recipe`, `training.finetuning_type`, and `training.retrieval.export.primary` have no per-flag form; supply them through `--spec` or `--spec-file`. ```python from nemo_automodel_plugin.schema import AutomodelJobInput job_suffix = uuid.uuid4().hex[:4] JOB_NAME = f"embed-nvdocs-{job_suffix}" OUTPUT_NAME = f"nemotron-3-embed-1b-tuned-{job_suffix}" spec = AutomodelJobInput( model=f"default/{MODEL_NAME}", dataset={"training": TRAINING_REF}, training={ "training_type": "sft", "recipe": "bi_encoder", "finetuning_type": "all_weights", "max_seq_length": 512, "retrieval": { "query_max_length": 512, "passage_max_length": 512, "train_n_passages": 5, "export": {"primary": "hf"}, }, }, schedule={"epochs": 1}, batch={"global_batch_size": 128, "micro_batch_size": 4}, optimizer={"learning_rate": 1e-5, "warmup_steps": 5, "weight_decay": 0.01}, parallelism={"num_gpus_per_node": 1}, output={"name": OUTPUT_NAME}, ) job = client.customization.automodel.jobs.create(spec=spec, workspace="default", name=JOB_NAME) print(f"Submitted {job.job.name}") print(f"Output model: {OUTPUT_NAME}") ``` ```python while True: status = client.jobs.get_status(name=job.job.name, workspace="default") clear_output(wait=True) print(status.model_dump_json(indent=2)) if status.status in ("completed", "failed", "cancelled", "error"): break time.sleep(30) if status.status != "completed": raise RuntimeError(f"Training finished with status: {status.status}") ``` ### Interpreting contrastive loss A healthy run drops sharply over the first 20–30% of training, then flattens toward a stable floor with validation loss tracking close behind. Validation loss rising while training loss falls is overfitting: cut epochs. Spikes or NaN mean the learning rate is too high. A curve that never moves points at the learning rate being too low or at the data itself. There is no target loss value, since the absolute number depends on batch size, negative count, and temperature. Loss is also not retrieval accuracy: it can keep improving while nDCG stagnates when the mined negatives are too easy. Stage 4 metrics are the ground truth. When iterating, sweep the learning rate at `5e-6`, `1e-5`, and `2e-5` with everything else fixed. If the best result lands on an endpoint, extend one step further in that direction. ## 5. Stage 3: deploy Retriever NIM 2.2.0 The base deployment serves the weights baked into the NIM image (`model_spec: {}`); the tuned deployment mounts the Automodel output entity. Both need `override_config.nimLegacy: false`, because NIM 2.x replaced `NIM_MODEL_NAME` and `NIM_MODEL_PATH` with their `NIM_ENGINE_*` equivalents and rejects the retired names. Scoring both models at once occupies two GPUs; the first image pull can take several minutes. Equivalent CLI: ```bash EXECUTOR='{"gpu":1,"image_name":"nvcr.io/nim/nvidia/nemotron-3-embed-1b","image_tag":"2.2.0","override_config":{"nimLegacy":false}}' nemo inference deployment-configs create embed-base-cfg --workspace default --exist-ok \ --engine nim --model-spec '{}' --executor-config "$EXECUTOR" \ --model-entity-id default/nemotron-3-embed-1b nemo inference deployment-configs create embed-tuned-cfg --workspace default --exist-ok \ --engine nim --executor-config "$EXECUTOR" \ --model-spec '{"model_namespace":"default","model_name":"nemotron-3-embed-1b-tuned"}' \ --model-entity-id default/nemotron-3-embed-1b-tuned nemo inference deployments create embed-base --workspace default --config embed-base-cfg \ --exist-ok --wait --timeout 1800 nemo inference deployments create embed-tuned --workspace default --config embed-tuned-cfg \ --exist-ok --wait --timeout 1800 nemo models get nemotron-3-embed-1b --workspace default nemo models get nemotron-3-embed-1b-tuned --workspace default ``` `retrieve-eval` only scores entities with a provider attached, so both `nemo models get` calls must show a non-empty `model_providers` list before Stage 4. Training exported Hugging Face weights to the fileset root; had it exported ONNX as primary, the tuned executor would also need `"additional_envs":{"NIM_ENGINE_MODEL_PATH":"alternates/hf"}`. ```python NIM_IMAGE = "nvcr.io/nim/nvidia/nemotron-3-embed-1b" NIM_TAG = "2.2.0" deploy_suffix = uuid.uuid4().hex[:4] BASE_CFG = f"embed-base-cfg-{deploy_suffix}" TUNED_CFG = f"embed-tuned-cfg-{deploy_suffix}" BASE_DEP = f"embed-base-{deploy_suffix}" TUNED_DEP = f"embed-tuned-{deploy_suffix}" executor = { "gpu": 1, "image_name": NIM_IMAGE, "image_tag": NIM_TAG, "override_config": {"nimLegacy": False}, } base_cfg = client.inference.deployment_configs.create( workspace="default", name=BASE_CFG, engine="nim", model_spec={}, executor_config=executor, model_entity_id=f"default/{MODEL_NAME}", ) tuned_cfg = client.inference.deployment_configs.create( workspace="default", name=TUNED_CFG, engine="nim", model_spec={"model_namespace": "default", "model_name": OUTPUT_NAME}, executor_config=executor, model_entity_id=f"default/{OUTPUT_NAME}", ) base_deployment = client.inference.deployments.create( workspace="default", name=BASE_DEP, config=base_cfg.name ) tuned_deployment = client.inference.deployments.create( workspace="default", name=TUNED_DEP, config=tuned_cfg.name ) print(base_deployment.name, tuned_deployment.name) ``` ```python for deployment in (BASE_DEP, TUNED_DEP): if not client.models.wait_for_status(deployment, "READY", workspace="default", timeout=2700): raise RuntimeError(f"{deployment} did not reach READY") for entity in (MODEL_NAME, OUTPUT_NAME): providers = client.models.retrieve(workspace="default", name=entity).model_providers print(entity, "providers:", providers) if not providers: raise RuntimeError(f"{entity} has no model_providers; retrieve-eval cannot score it") ``` ## 6. Serving sanity (`/v1/embeddings`) One request confirms the endpoint answers and returns 2048 dimensions. Pass `input_type` as `query` or `document` to match how the text will be used at retrieval time. Equivalent CLI: ```bash nemo inference gateway provider post v1/embeddings embed-tuned --workspace default \ --body '{"model":"default/nemotron-3-embed-1b-tuned","input":["What does NVLink connect?"],"input_type":"query"}' ``` ```python resp = client.inference.gateway.provider.post( "v1/embeddings", name=TUNED_DEP, workspace="default", body={ "model": f"default/{OUTPUT_NAME}", "input": ["What does NVLink connect?"], "input_type": "query", }, ) print("dim", len(resp["data"][0]["embedding"])) ``` ## 7. Stage 4: `retrieve-eval` on the frozen `eval_beir` split `retrieve-eval` embeds the corpus with each model, ranks the queries, and reports nDCG\@k, Recall\@k, Precision\@k, and MAP\@k for the target and the baseline. Pass the same artifacts fileset every run. Equivalent CLI: ```bash nemo evaluator retrieve-eval submit --workspace default --spec '{ "dataset": "default/nvdocs-embed-artifacts", "target": "default/nemotron-3-embed-1b-tuned", "baseline": "default/nemotron-3-embed-1b", "k": [1, 5, 10, 100] }' nemo jobs watch --workspace default nemo jobs results list --workspace default --output json ``` `target`, `baseline`, and `k` have no per-flag form; supply them through `--spec` or `--spec-file`. ```python import shutil import subprocess eval_spec = { "dataset": TRAINING_REF, "target": f"default/{OUTPUT_NAME}", "baseline": f"default/{MODEL_NAME}", "k": [1, 5, 10, 100], } cmd = [ shutil.which("nemo") or "nemo", "evaluator", "retrieve-eval", "submit", "--workspace", "default", "--spec", json.dumps(eval_spec), ] proc = subprocess.run(cmd, check=False, capture_output=True, text=True) if proc.returncode != 0: raise RuntimeError(f"retrieve-eval submit failed:\n{proc.stderr}") eval_job_name = json.loads(proc.stdout[proc.stdout.index("{") :])["name"] print("retrieve-eval job:", eval_job_name) while True: status = client.jobs.get_status(name=eval_job_name, workspace="default") print("eval status:", status.status) if status.status in ("completed", "failed", "cancelled", "error"): break time.sleep(30) if status.status != "completed": raise RuntimeError(f"retrieve-eval finished with status: {status.status}") for row in client.jobs.results.list(eval_job_name, workspace="default").data: print(json.dumps(row.model_dump(), default=str, indent=2)[:4000]) ``` ## Interpreting results An example result after finetuning for 3 epochs on NVDocs SDG dataset and evaluating on 20,909-query eval\_beir split: ``` Model: base - nDCG@10: 0.56713 - Recall@10: 0.63753 - Precision@10: 0.13192 Model: fine-tuned - nDCG@10: 0.61696 (+9%) - Recall@10: 0.69999 (+10%) - Precision@10: 0.14807 (+12%) ``` Compare the fine-tuned model against the base checkpoint in the same evaluation. The eval split, corpus size, domain, and mining depth all move the scale. ## Hyperparameters Learning rate is the most sensitive parameter, followed by epochs, warmup, and batch size. | Parameter | API field | Value | | ------------------ | --------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | | Learning rate | `optimizer.learning_rate` | `1e-5`, swept at half and double | | Warmup | `optimizer.warmup_steps` | `5`, or 5–10% of total steps | | Weight decay | `optimizer.weight_decay` | `0.01` | | Epochs | `schedule.epochs` | `1` for 10,000 rows or more, `2` for a few thousand | | Global batch | `batch.global_batch_size` | `128` | | Micro batch | `batch.micro_batch_size` | `4` | | Sequence length | `training.max_seq_length`, `training.retrieval.query_max_length`, `training.retrieval.passage_max_length` | `512` | | Passages per query | `training.retrieval.train_n_passages` | `5`, one positive and four mined negatives | Batch size sets how many gradient steps an epoch takes rather than how many negatives each query sees, so smaller datasets benefit from a smaller global batch. When you scale data, adding documents helps more than generating more queries per document: new documents bring new vocabulary and retrieval patterns, while extra queries cover the same passage from another angle. ## Troubleshooting * **`neg_doc must contain at least 1 document`**: Stage 1 ran without mining. Re-run `retrieval-prepare` with `enable_mining: true` and a fileset-backed `model`. * **CUDA out of memory during mining**: lower `mining.query_embedding_batch_size` and `mining.document_embedding_batch_size`. * **Empty train split**: the corpus is too small. Generating your own data needs 50 documents at minimum and 500 or more for good coverage. * **`retrieve-eval` reports no `model_providers`**: the NIM is not serving the entity yet. Wait for `READY` and for the gateway to attach the provider. ## Clean up Each deployment holds a GPU. Release them once evaluation finishes; the configs can stay for the next run. ```bash nemo inference deployments delete embed-base --workspace default nemo inference deployments delete embed-tuned --workspace default ``` ## Next steps * [Generate retrieval training data](/documentation/design-synthetic-data/tutorials/retrieval-generate) from your own corpus * [Train on SPECTER or triplet JSONL](/documentation/customizer-reference/tutorials/embedding-triplet-dataset) * [Retrieval SDG reference](/documentation/design-synthetic-data/retrieval-sdg) * [Job status](/documentation/customizer-reference/manage-customization-jobs/get-job-status)