Quickstart: retriever CLI

Use retriever ingest and retriever query for product-facing workflows.

Quick start¶

Local ingest embeds on the local GPU when --embed-invoke-url is unset. Install the [local] extra before you run the default retriever ingest or retriever ingest batch commands:

pip install "nemo-retriever[local]"

If you installed the base package for Remote NIM with no local GPU, keep that install and pass --embed-invoke-url instead. Refer to Route ingest to hosted or self-hosted NIM endpoints.

Ingest a PDF locally¶

From a clone of this repository, ./data/multimodal_test.pdf is a valid first-run input. If you installed from PyPI, pass a PDF file that you supply.

retriever ingest ./data/multimodal_test.pdf

Then query the default LanceDB table:

retriever query "What is in this document?"

By default, local ingest auto-detects supported input formats and writes to lancedb/nemo-retriever; retriever query reads from the same table. Use explicit high-level options when a task needs behavior beyond the current ingest defaults.

The plain retriever query examples below apply to local and batch ingest output written to LanceDB. Use retriever query service to query a Retriever service.

Ingest a larger corpus with batch mode¶

Replace /path/to/your/pdfs with a directory of PDF files that you supply. The repository and the PyPI package do not include a pdf_corpus dataset.

retriever ingest batch /path/to/your/pdfs \
  --profile fast-text \
  --pdf-extract-workers 4 \
  --embed-workers 2

Batch mode exposes Ray runtime and batch tuning flags such as --ray-address, --pdf-extract-workers, --ocr-workers, and --embed-workers.

Ingest through a Retriever service¶

Replace /path/to/your/pdfs with a directory of PDF files that you supply.

retriever ingest service /path/to/your/pdfs \
  --service-url http://localhost:7670 \
  --service-concurrency 8

Use --service-api-token or NEMO_RETRIEVER_API_TOKEN when the service requires a bearer token. Service ingest does not expose --lancedb-uri; the service configures its vector database. Query the service with:

retriever query service "What is in this corpus?" \
  --service-url http://localhost:7670

Start a local service with VectorDB¶

Use retriever service start --launch-vectordb to run a local service with a supervised VectorDB child on 127.0.0.1:7671. The child uses nim_endpoints.embed_invoke_url when configured. Otherwise, it uses local Hugging Face embedding when local_models.enabled and local_models.embed.enabled are both true. The command waits for VectorDB readiness and terminates the child when the service exits. If the child does not exit promptly, the service forcefully stops it. You can set the same behavior in YAML with vectordb.launch_on_start: true and a loopback vectordb.vectordb_url.

The child inherits credentials from the service environment. Set NVIDIA_API_KEY or NGC_API_KEY for remote embedding, and set NRL_INTERNAL_VDB_TOKEN or NRL_INTERNAL_VDB_TOKEN_FILE for the VectorDB internal credential. Do not place credentials in the service YAML.

For a fully local deployment, use a CUDA-capable host and install the service and local extras. Install the multimedia extra when you ingest audio or video.

pip install "nemo-retriever[service,local]"
scripts/launch_local_service_with_vectordb.sh \
  nemo_retriever/examples/retriever-service.local.yaml

The example configuration leaves NIM endpoints unset and uses local Hugging Face models. The launcher validates /v1/health on the service and VectorDB, then keeps both processes running until you stop it with Ctrl+C.

retriever service start --config my-retriever-service.yaml --launch-vectordb

Use the command above when nim_endpoints.embed_invoke_url is configured. Omit the flag to use an existing VectorDB. Helm continues to deploy VectorDB as a separate pod.

If VectorDB exits during startup or does not become ready, inspect the VectorDB output in the terminal that started the service. Verify the VectorDB configuration, embedding model setup and credentials, writable LanceDB directory, and that port 7671 is available.

Route ingest to hosted or self-hosted NIM endpoints¶

export NVIDIA_API_KEY=nvapi-...

retriever ingest ./data/multimodal_test.pdf \
  --page-elements-invoke-url https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-page-elements-v3 \
  --ocr-invoke-url https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-ocr-v2 \
  --table-structure-invoke-url https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-table-structure-v1 \
  --embed-invoke-url https://integrate.api.nvidia.com/v1/embeddings \
  --embed-model-name nvidia/llama-nemotron-embed-1b-v2

NVIDIA_API_KEY is required only when those URLs point at hosted build.nvidia.com endpoints. NGC_API_KEY is used separately when pulling or running self-hosted NIM containers.

For NVIDIA inference hub rerank models that expose the Cohere-style rerank route, pass the full /v1/rerank URL and the model name shown in the hub snippet:

export NGC_INFERENCE_API_KEY=...

retriever query "What is in this document?" \
  --embed-invoke-url https://integrate.api.nvidia.com/v1/embeddings \
  --embed-model-name nvidia/llama-nemotron-embed-1b-v2 \
  --reranker-invoke-url https://inference-api.nvidia.com/v1/rerank \
  --reranker-model-name nvidia/nvidia/llama-3.2-nv-rerankqa-1b-v2 \
  --reranker-api-key-env NGC_INFERENCE_API_KEY

Query result controls¶

Both retriever query and retriever query service return compact JSON hits with source, page_number, and text. Use --candidate-k, --page-dedup, and --content-types to control how results are selected after vector retrieval:

retriever query "annual revenue by region" \
  --top-k 5 \
  --candidate-k 40 \
  --content-types table

--top-k is the final number of results to return after filtering and deduplication. --candidate-k is the number of raw results to retrieve from LanceDB or the Retriever service before filtering, page deduplication, and final truncation. If omitted, the candidate pool is the same size as --top-k. Set --candidate-k larger than --top-k when page deduplication or content-type filtering might remove too many of the nearest retrieved rows. It must always be greater than or equal to --top-k.

Page deduplication and content-type filtering are applied after vector retrieval, preserving retriever ranking order and truncating the final output to --top-k. Local and batch ingest record the canonical embedding model on the LanceDB table, and non-service query uses that model automatically. Use --embed-model-name only as an explicit override or when querying a legacy or third-party table without model metadata. Endpoint URLs and provider prefixes remain runtime configuration, so continue to pass --embed-invoke-url and --embed-model-provider-prefix when the selected model must be routed remotely. For example, a table can store the canonical model nvidia/llama-nemotron-embed-vl-1b-v2 while a LiteLLM-routed request uses nvidia/nvidia/llama-nemotron-embed-vl-1b-v2. The endpoint and routing prefix are intentionally not persisted on the table.

--content-types accepts comma-separated content types such as text, table, chart, image, and infographic. images is accepted as an alias for captioned image rows emitted by ingest. This option filters by content-type metadata only; it does not filter by source, page, or other metadata predicates. Hits with missing or unknown content-type metadata are excluded while --content-types is active. In service mode, results must include content-type metadata to match this filter. Default display values in the JSON output are not used for content-type matching.

Agentic retrieval¶

--agentic swaps the single dense pass for an LLM-driven ReAct loop: the agent issues several retrieval sub-queries, fuses the candidates, and selects a final ranking. It searches the same LanceDB table built by retriever ingest, so it is a drop-in alternative to standard retrieval.

By default, agentic retrieval runs the agent LLM in process with local vLLM and nemotron-8b (nvidia/Llama-3.1-Nemotron-Nano-8B-v1). This requires a CUDA GPU host and the local extras installed. Provide --agentic-invoke-url when you want a custom model or a separately hosted OpenAI-compatible endpoint.

# default local vLLM agent LLM: nemotron-8b
retriever query "how does the ingestion pipeline handle tables?" \
  --agentic

# custom/self-hosted model through an OpenAI-compatible endpoint
retriever query "summarize the deployment options" \
  --agentic \
  --agentic-llm-model custom-remote-model \
  --agentic-invoke-url http://localhost:9000/v1/chat/completions \
  --embed-invoke-url http://localhost:8000/v1 \
  --agentic-react-max-steps 5

Agentic mode returns the agent's ranked documents as JSON, with the same hit fields as the dense path (text, metadata, source, page_number, and related) plus doc_id, rank, and the stage that produced the ranking (final_results, rrf, or selection_agent). Hit fields are rehydrated at the end of the loop from the retrieval hop that returned the document, so a document the agent named without retrieving it reports null hit fields. It reuses the same --top-k, --lancedb-uri, --table-name, --embed-invoke-url, and --embed-model-name options as standard retrieval. Agentic retrieval uses the selected table's model automatically when --embed-model-name is omitted.

How it works. Each agentic query runs Query -> ReActAgentOperator -> (RRF fusion) -> SelectionAgentOperator -> ranked results:

ReActAgentOperator runs the per-query ReAct loop; every retrieve tool call delegates to the standard Retriever, so the agent searches the same vector DB and embedding config as dense retrieval.
RRFAggregatorOperator fuses candidates from the loop's multiple searches with reciprocal rank fusion.
SelectionAgentOperator runs a final LLM selection pass over the fused set and emits the ranked document IDs, which are then rehydrated into full hits.

Agentic-only knobs (apply only with --agentic):

--agentic-llm-model — local profile alias/model ID when no invoke URL is provided (nemotron-8b by default; super-49b also supported), or the remote model ID when --agentic-invoke-url is provided.
--agentic-local-tensor-parallel-size (default 1) — vLLM tensor_parallel_size for the in-process agent LLM. Use 2+ with matching CUDA_VISIBLE_DEVICES for multi-GPU local profiles (for example super-49b). Ignored when --agentic-invoke-url is set. When the first tensor_parallel_size CUDA-visible GPUs are not NVLink-connected (typical dual-GPU PCIe workstations), tensor-parallel startup automatically sets NCCL_NVLS_ENABLE=0 and TORCH_SYMM_MEM_DISABLE_MULTICAST=1, because NVLink multicast collectives abort vLLM startup there; set either variable yourself to override. Detection is scoped to that TP device group, not the whole host or extra visible GPUs outside the shard.
--agentic-invoke-url — OpenAI-compatible chat-completions endpoint for the agent LLM. Providing it routes agent LLM calls to that remote endpoint; omit it to run the in-process local model.
--agentic-llm-client (optional) — LLM client that builds the agent LLM. Defaults to callable. It drives the in-process adapter when --agentic-invoke-url is omitted, and the shared chat-completions HTTP client when it is set.
--agentic-reasoning-effort (default high) — reasoning_effort forwarded on OpenAI-compatible agentic LLM calls; ignored by the local adapter.
--agentic-react-max-steps (default 50) — maximum ReAct loop iterations.
--agentic-text-truncation (default 0) — max characters of each candidate shown to the agent; 0 disables truncation.
--agentic-temperature (default: unset) — sampling temperature for agent LLM calls; omit to use the endpoint/model default (0.0 = greedy). Local and non-NVIDIA OpenAI-compatible endpoints allow up to 2.0; NVIDIA-hosted endpoints allow up to 1.0.