> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server.

# Retrieval SDG

> Generate and prepare retrieval training data with the Nemotron Stage 0 and Stage 1 recipes.

NeMo Platform exposes Nemotron embed/rerank Stage 0 (`sdg`) and Stage 1
(`prep`) as dedicated Data Designer jobs. They wrap
`data-designer-retrieval-sdg` and do **not** go through
`nemo data-designer create`.

## Prerequisites

* A text corpus published as a fileset in the job workspace, or an `hf://`
  dataset URI.
* An Inference Gateway provider for the Stage 0 chat model and, when different,
  an embedding provider.
* For GPU mining, a platform model entity with an attached fileset containing
  the encoder and tokenizer.

## Generate (Stage 0, CPU)

```bash
nemo data-designer retrieval-generate --workspace default --spec '{
  "corpus": "default/my-docs",
  "provider": "default/nvidia-build",
  "artifact_extraction_model": "nvidia/nemotron-3-nano-30b-a3b",
  "qa_generation_model": "nvidia/nemotron-3-nano-30b-a3b",
  "quality_judge_model": "nvidia/nemotron-3-nano-30b-a3b",
  "embed_model": "nvidia/nemotron-3-embed-1b"
}'
```

Pass the chat and embedding model names served by Inference Gateway. All four
model roles call IGW through `provider` (workspace/name), with optional
`chat_provider` / `embed_provider` overrides. Do not pass raw `NVIDIA_API_KEY`.

Equivalent helper:

```bash
nemo data-designer retrieval generate \
  --corpus default/my-docs \
  --provider default/nvidia-build \
  --chat-model nvidia/nemotron-3-nano-30b-a3b \
  --embed-model nvidia/nemotron-3-embed-1b
```

Output fileset includes Q\&A JSONL and `generation_result.json`.

Preview without a full job:

```bash
nemo data-designer retrieval-preview --spec '{"generate":{"corpus":"default/my-docs","provider":"default/nvidia-build","artifact_extraction_model":"nvidia/nemotron-3-nano-30b-a3b","qa_generation_model":"nvidia/nemotron-3-nano-30b-a3b","quality_judge_model":"nvidia/nemotron-3-nano-30b-a3b","embed_model":"nvidia/nemotron-3-embed-1b"},"num_records":1}'
```

## Prepare (Stage 1)

Conversion (CPU) produces `eval_beir/` and training JSON. GPU mining is off by
default. Skip SDG by pointing `sdg_input` at a fileset containing an existing
`generation_result.json`, or at
`hf://nvidia/Retrieval-Synthetic-NVDocs-v1@<revision>`.

```bash
nemo data-designer retrieval-prepare --spec '{
  "sdg_input": "default/stage0-out"
}'
```

Enable mining (GPU, \~40GB) with `"enable_mining": true`. The `model` field identifies
a platform model entity whose fileset is downloaded by `nmp-customizer-tasks` into
the shared job storage. `nmp-automodel-training` then loads the encoder and tokenizer
from that local directory with Hugging Face networking disabled. Conversion-only
prepare stays on `nmp-cpu-tasks`.

All retrieval steps use the same container-backed execution profile so they share job
storage across generation, conversion, model staging, and mining. Configure it with
`data_designer.job_executor_profile`, or pass `--profile` for an individual job. The
local platform uses `gpu`; Kubernetes uses `default`.

## Chain generate then prepare

```bash
nemo data-designer retrieval-run --spec '{
  "generate": {
    "corpus": "default/my-docs",
    "provider": "default/nvidia-build",
    "artifact_extraction_model": "nvidia/nemotron-3-nano-30b-a3b",
    "qa_generation_model": "nvidia/nemotron-3-nano-30b-a3b",
    "quality_judge_model": "nvidia/nemotron-3-nano-30b-a3b",
    "embed_model": "nvidia/nemotron-3-embed-1b"
  },
  "prepare": {}
}'
```

This is jobs-service multi-step execution (CPU then optional GPU), not Data Designer
in-config workflow chaining.

## SDK

```python
from nemo_platform import NeMoPlatform
from nemo_data_designer_plugin.jobs.retrieval_spec import RetrievalGenerateJobConfig

client = NeMoPlatform(base_url="http://localhost:8080", workspace="default")

client.data_designer.retrieval_generate(
    RetrievalGenerateJobConfig(
        corpus="default/my-docs",
        provider="default/nvidia-build",
        artifact_extraction_model="nvidia/nemotron-3-nano-30b-a3b",
        qa_generation_model="nvidia/nemotron-3-nano-30b-a3b",
        quality_judge_model="nvidia/nemotron-3-nano-30b-a3b",
        embed_model="nvidia/nemotron-3-embed-1b",
    )
)
```

## Next Steps

* See the [Data Designer CLI](/documentation/design-synthetic-data/cli) for configuration and command details.
* Use the Stage 1 `training.jsonl` artifact as input to an embedding or
  reranking customization job.